Explicit Segmentation Research Articles

Enhancing the Recognition of Handwritten Arabic Characters through Hybrid Convolutional and Bidirectional Recurrent Neural Network Models

Handwritten Arabic character recognition remains a challenging task in pattern recognition due to the inherent complexities of the cursive script and visual similarities between characters. While deep learning techniques have demonstrated promising results in this domain, further enhancements to the model architecture can drive even greater performance improvements. This study introduces a hybrid deep learning approach that combines Convolutional Neural Networks (CNNs) with Bidirectional Recurrent Neural Networks, specifically Bidirectional Long Short-Term Memory (Bi-LSTM) and Bidirectional Gated Recurrent Units (Bi-GRU). By leveraging the strengths of both convolutional and recurrent neural network components, the proposed models are able to effectively capture spatial features as well as model the temporal dynamics and contextual relationships present in handwritten Arabic text. Experiments conducted on the AHCD and Hijjaa benchmark datasets show that the CNN-Bi-GRU framework achieved state-of-the-art accuracy rates of 97.05% and 91.78% respectively, outperforming previous deep learning-based methods. These results demonstrate the significant performance gains that can be achieved by integrating specialized temporal modeling and contextual representation capabilities into the handwriting recognition pipeline, without the need for explicit segmentation. The findings of this research represent a crucial advancement in the continued development of sophisticated and precise deep learning systems for Arabic handwriting recognition, with broad applications across domains that rely on efficient text extraction from handwritten documents.

Consistency label-activated region generating network for weakly supervised medical image segmentation

The current methods of auto-segmenting medical images are limited due to insufficient and ambiguous pathonmorphological labeling. In clinical practice, rough classification labels (such as disease or normal) are more commonly used than precise segmentation masks. However, there is still much to be explored regarding utilizing these weak clinical labels to accurately determine the lesion mask and guide medical image segmentation. In this paper, we proposed a weakly supervised medical image segmentation model to directly generate the lesion mask through a class activation map (CAM) guided cycle-consistency label-activated region transferring network. Cycle-consistency enforces that the mappings between the two domains should be reversible, which ensures that the original image can be reconstructed from the translated image. We developed a complementary branches fusion module to address the issue of blurry boundaries in CAM-guided segmentation. The complementary branch preserves the original semantic information of the non-lesion region and perfectly fuses the transferred feature of the lesion region with a complementary mask-constrained fake image generation process to clear the boundary of the lesion and non-lesion regions. This module allows the class transformation to focus solely on the label-activated region, resulting in more explicit segmentation. This model can accurately identify different region of medical images at the pixel-level while preserving the overall semantic structure semantion. It organizes disease labels and corresponding regions during image synthesis. Our method utilizes a joint discrimination strategy that significantly enhances the precision of the produced lesion mask. Extensive experiments of the proposed method on BraTs, ISIC and COVID-19 datasets demonstrate superior performance over existing state-of-the-art methods. The code and datasets are available at: https://github.com/mlcb-jlu/MedImgSeg.

Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation

We propose a novel unsupervised method to autoregressively generate videos from a single frame and a sparse motion input. Our trained model can generate unseen realistic object-to-object interactions. Although our model has never been given the explicit segmentation and motion of each object in the scene during training, it is able to implicitly separate their dynamics and extents. Key components in our method are the randomized conditioning scheme, the encoding of the input motion control, and the randomized and sparse sampling to enable generalization to out of distribution but realistic correlations. Our model, which we call YODA, has therefore the ability to move objects without physically touching them. Through extensive qualitative and quantitative evaluations on several datasets, we show that YODA is on par with or better than state of the art video generation prior work in terms of both controllability and video quality.

An approach for handwritten Chinese text recognition unifying character segmentation and recognition

Text line recognition methods can be categorized into explicit segmentation based and implicit segmentation based ones. Explicit segmentation based methods require character-level annotation during training, while implicit segmentation based methods, trained on line-level annotated data, face alignment drift challenges. Though some methods have been proposed to address these challenges using weakly supervised object detection, they often rely on cumbersome pseudo-box generation processes and complex decoding. In this paper, we propose a unified framework to overcome these challenges, achieving high accuracy in text recognition and character segmentation. To eliminate the need of character-level annotated real text line data in training, we introduce a novel training paradigm that utilizes character-level annotated synthetic data and line-level annotated real data jointly. For synthetic data, candidate characters are explicitly aligned with labeled characters to generate hard labels for supervising model training. For real data, implicit alignment is produced by Connectionist Temporal Classification (CTC) mapping to provide soft labels for weakly-supervised model training. And for inference, we propose two decoding strategies leveraging the advantages of Non-Maximum Suppression (NMS) and CTC decoding. Extensive experiments on benchmark datasets demonstrate the superior performance of our method in text recognition and character localization, even with minimal amounts of character-level annotated line data.

MD-Cave: An Immersive Visualization Workbench for Radiologists.

The MD-Cave is an immersive analytics system that provides enhanced stereoscopic visualizations to support visual diagnoses performed by radiologists. The system harnesses contemporary paradigms in immersive visualization and 3D interaction, which are better suited for investigating 3D volumetric data. We retain practicality through efficient utilization of desk space and comfort for radiologists in terms of frequent long duration use. MD-Cave is general and incorporates: (1) high resolution stereoscopic visualizations through a surround triple-monitor setup, (2) 3D interactions through head and hand tracking, (3) and a general framework that supports 3D visualization of deep-seated anatomical structures without the need for explicit segmentation algorithms. Such a general framework expands the utility of our system to many diagnostic scenarios. We have developed MD-Cave through close collaboration and feedback from two expert radiologists who evaluated the utility of MD-Cave and the 3D interactions in the context of radiological examinations. We also provide evaluation of MD-Cave through case studies performed by an expert radiologist and concrete examples on multiple real-world diagnostic scenarios, such as pancreatic cancer, shoulder-CT, and COVID-19 Chest CT examination.

ExhibitXplorer: Enabling Personalized Content Delivery in Museums Using Contextual Geofencing and Artificial Intelligence

In recent years, there has been an increasing demand for personalized experiences in various domains, including the cultural and educational sectors. Museums, as custodians of art, history, and scientific knowledge, are seeking innovative ways to engage their visitors and provide tailored content that enhances their understanding and appreciation of the exhibits. This article presents ExhibitXplorer, a distributed architecture service that leverages geofencing, artificial intelligence, and microservices to enable personalized content delivery in museums. By combining implicit and explicit segmentation of museum visitors and utilizing the GPT API for content generation, ExhibitXplorer offers a dynamic experience to different visitor segments, including researchers, students, casual visitors, and children. The system utilizes push notifications triggered by visitor location changes, allowing seamless delivery of personalized information both indoors and outdoors. Tests were conducted to evaluate the user experience of visitors to an outdoor ethnographic museum. The results showed that 55% of the test participants were satisfied and 45% very satisfied with the way personalized content was delivered.

Mice and primates use distinct strategies for visual segmentation.

The rodent visual system has attracted great interest in recent years due to its experimental tractability, but the fundamental mechanisms used by the mouse to represent the visual world remain unclear. In the primate, researchers have argued from both behavioral and neural evidence that a key step in visual representation is 'figure-ground segmentation', the delineation of figures as distinct from backgrounds. To determine if mice also show behavioral and neural signatures of figure-ground segmentation, we trained mice on a figure-ground segmentation task where figures were defined by gratings and naturalistic textures moving counterphase to the background. Unlike primates, mice were severely limited in their ability to segment figure from ground using the opponent motion cue, with segmentation behavior strongly dependent on the specific carrier pattern. Remarkably, when mice were forced to localize naturalistic patterns defined by opponent motion, they adopted a strategy of brute force memorization of texture patterns. In contrast, primates, including humans, macaques, and mouse lemurs, could readily segment figures independent of carrier pattern using the opponent motion cue. Consistent with mouse behavior, neural responses to the same stimuli recorded in mouse visual areas V1, RL, and LM also did not support texture-invariant segmentation of figures using opponent motion. Modeling revealed that the texture dependence of both the mouse's behavior and neural responses could be explained by a feedforward neural network lacking explicit segmentation capabilities. These findings reveal a fundamental limitation in the ability of mice to segment visual objects compared to primates.

Autonomous Machinery Management for Supervisory Risk Control Using Particle Swarm Optimization

Safe navigation for maritime autonomous surface ships (MASS) is a challenging task, and generally highly dependent on effective collaboration between multiple sub-systems in environments with various levels of uncertainty. This paper presents a novel methodology combining risk-based optimal control and path following with autonomous machinery management (AMM) for MASS navigation and supervisory risk control. Specifically, a risk-aware particle swarm optimization (PSO) scheme utilizes “time-to-grounding” predictions based on weather data and electronic navigational charts (ENC) to simultaneously control both the ship’s motion as well as the machinery system operation (MSO) mode during transit. The proposed autonomous navigation system (ANS) is comprised of an online receding horizon control that uses a PSO approach from previous works, which produces a dynamic risk-aware path with respect to grounding obstacles from a pre-planned MASS path, subsequently given as the input to a line-of-sight guidance controller for path following. Moreover, the MSO mode of the AMM system is simultaneously selected and assigned to explicit segments along the risk-aware path throughout the receding horizon, which effectively introduces into the optimization scheme an additional safety layer as well as another dimension for risk or resource minimization. The performance of the resulting ANS is demonstrated and verified through simulations of a challenging scenario and human assessment of the generated paths. The results show that the optimized paths are more efficient and in line with how human navigators would maneuver a ship close to nearby grounding obstacles, compared to the optimized paths of selected previous works.

Visual News Ticker Surveillance Approach from Arabic Broadcast Streams

The news ticker is a common feature of many different news networks that display headlines and other information. News ticker recognition applications are highly valuable in e-business and news surveillance for media regulatory authorities. In this paper, we focus on the automatic Arabic Ticker Recognition system for the Al-Ekhbariya news channel. The primary emphasis of this research is on ticker recognition methods and storage schemes. To that end, the research is aimed at character-wise explicit segmentation using a semantic segmentation technique and words identification method. The proposed learning architecture considers the grouping of homogeneous-shaped classes. This incorporates linguistic taxonomy in a unified manner to address the imbalance in data distribution which leads to individual biases. Furthermore, experiments with a novel Arabic News Ticker (Al-ENT) dataset that provides accurate character-level and character components-level labeling to evaluate the effectiveness of the suggested approach. The proposed method attains 96.5%, outperforming the current state-of-the-art technique by 8.5%. The study reveals that our strategy improves the performance of low-representation correlated character classes.

Combination of explicit segmentation with Seq2Seq recognition for fine analysis of children handwriting

We consider the task of analyzing children handwriting in the context of a dictation task. The objective is to detect orthographic and phonological errors. To achieve this goal, we extend an existing handwriting analysis engine, based on an explicit segmentation of the handwritten input, originally developed for children copying exercises. We present a new approach, based on the combination of this analysis engine with a deep learning word recognition approach in order to improve both the recognition and segmentation performance. Explicit segmentation needs prior knowledge, and the deep network recognition predictions are a reliable approximation of the ground truth which can guide the analysis process. We propose to combine multiple prior knowledge strategies to further improve the analysis performance. Furthermore, we exploit the deep network approximate implicit segmentation to optimize the existing analysis process in terms of complexity.

DeepOrganoid: A brightfield cell viability model for screening matrix-embedded organoids.

High-throughput viability screens are commonly used in the identification and development of chemotherapeutic drugs. These systems rely on the fidelity of the cellular model systems to recapitulate the drug response that occurs in vivo. In recent years, there has been an expansion in the utilization of patient-derived materials as well as advanced cell culture techniques, such as multi-cellular tumor organoids, to further enhance the translational relevance of cellular model systems. Simple quantitative analysis remains a challenge, primarily due to the difficulties of robust image segmentation in heterogenous 3D cultures. However, explicit segmentation is not required with the advancement of deep learning, and it can be used for both continuous (regression) or categorical classification problems. Deep learning approaches are additionally benefited by being fully data-driven and highly automatable, thus they can be established and run with minimal to no user-defined parameters. In this article, we describe the development and implementation of a regressive deep learning model trained on brightfield images of patient-derived organoids and use the terminal viability readout (CellTiter-Glo) as training labels. Ultimately, this has led to the generation of a non-invasive and label-free tool to evaluate changes in organoid viability.

Renal Cancer Detection: Fusing Deep and Texture Features from Histopathology Images.

Histopathological images contain morphological markers of disease progression that have diagnostic and predictive values, with many computer-aided diagnosis systems using common deep learning methods that have been proposed to save time and labour. Even though deep learning methods are an end-to-end method, they perform exceptionally well given a large dataset and often show relatively inferior results for a small dataset. In contrast, traditional feature extraction methods have greater robustness and perform well with a small/medium dataset. Moreover, a texture representation-based global approach is commonly used to classify histological tissue images expect in explicit segmentation to extract the structure properties. Considering the scarcity of medical datasets and the usefulness of texture representation, we would like to integrate both the advantages of deep learning and traditional machine learning, i.e., texture representation. To accomplish this task, we proposed a classification model to detect renal cancer using a histopathology dataset by fusing the features from a deep learning model with the extracted texture feature descriptors. Here, five texture feature descriptors from three texture feature families were applied to complement Alex-Net for the extensive validation of the fusion between the deep features and texture features. The texture features are from (1) statistic feature family: histogram of gradient, gray-level cooccurrence matrix, and local binary pattern; (2) transform-based texture feature family: Gabor filters; and (3) model-based texture feature family: Markov random field. The final experimental results for classification outperformed both Alex-Net and a singular texture descriptor, showing the effectiveness of combining the deep features and texture features in renal cancer detection.

Improving robustness of automatic cardiac function quantification from cine magnetic resonance imaging using synthetic image data

Although having been the subject of intense research over the years, cardiac function quantification from MRI is still not a fully automatic process in the clinical practice. This is partly due to the shortage of training data covering all relevant cardiovascular disease phenotypes. We propose to synthetically generate short axis CINE MRI using a generative adversarial model to expand the available data sets that consist of predominantly healthy subjects to include more cases with reduced ejection fraction. We introduce a deep learning convolutional neural network (CNN) to predict the end-diastolic volume, end-systolic volume, and implicitly the ejection fraction from cardiac MRI without explicit segmentation. The left ventricle volume predictions were compared to the ground truth values, showing superior accuracy compared to state-of-the-art segmentation methods. We show that using synthetic data generated for pre-training a CNN significantly improves the prediction compared to only using the limited amount of available data, when the training set is imbalanced.

Recognition of Visual Arabic Scripting News Ticker From Broadcast Stream

News ticker recognition is a vital area of research due to its applications such as information analysis, opinion mining and language translation for media regulatory authorities. Without automated systems, manual anatomizing is difficult. In this paper, we focus on the automatic Arabic and Urdu news ticker recognition system. It mainly consists of ticker segmentation and text recognition to generate textual data for various online services. Our work investigates character-wise explicit segmentation and syntactical models with Kufi and Nastaleeq fonts. Various network models anticipate learning of deep representations by homogenizing the classes regardless of inter-symbol correlations and linguistic taxonomy. The proposed learning model incorporates fairness by maximizing the balance among sensitive features of characters in a unified manner. Furthermore, we demonstrate the efficiency of the proposed model by carrying out experiments using customized news tickers datasets with accurate character-level and component-level labeling. Moreover, our method is evaluated on a challenging Urdu Printed Text Images (UPTI) dataset that only provides ligature based annotations. The proposed method attains 98.36%, outperforms the current state of the art method. Ablation investigations show that our technique enhances the performance of character classes with low symbol frequencies.

Audio Matters in Video Super-Resolution by Implicit Semantic Guidance

Video super-resolution (VSR) aims to use multiple consecutive low-resolution frames to recover the corresponding high-resolution frames. However, existing VSR methods only consider videos as image sequences, ignoring another essential timing informationaudio, while in fact, there is a semantic link between audio and vision, and extensive studies have shown that audio can provide supervisory information in visual networks. Meanwhile, the addition of semantic priors has been proven to be effective in super-resolution (SR) tasks, but a pretrained segmentation network is required to obtain semantic segmentation maps. By contrast, audio as the information contained in the video itself can be directly used. Therefore, in this study, we propose a novel and pluggable multiscale audiovisual fusion (MS-AVF) module to enhance VSR performance by exploiting the relevant audio information, which can be regarded as implicit semantic guidance compared with the kind of explicit segmentation priors. Specifically, we first fuse audiovisual features on the semantic feature maps of different granularities of the target frames, and then through a top-down multiscale fusion approach, feedback high-level semantics to the underlying global visual features layer by layer, thereby providing effective audio implicit semantic guidance for VSR. Experimental results show that audio can further improve the VSR effect. Moreover, by visualizing the learned attention mask, the proposed end-to-end model can automatically learn potential audiovisual semantic links, especially improving the accuracy and effectiveness of the SR of sound sources and their surrounding regions.

Do we really need a segmentation step in heart sound classification algorithms?

Cardiac auscultation is the key screening procedure to detect and identify cardiovascular diseases (CVDs). One of many steps to automatically detect CVDs using auscultation, concerns the detection and delimitation of the heart sound boundaries, a process known as segmentation. Whether to include or not a segmentation step in the signal classification pipeline is nowadays a topic of discussion. Up to our knowledge, the outcome of a segmentation algorithm has been used almost exclusively to align the different signal segments according to the heartbeat. In this paper, the need for a heartbeat alignment step is tested and evaluated over different machine learning algorithms, including deep learning solutions. From the different classifiers tested, Gate Recurrent Unit (GRU) Network and Convolutional Neural Network (CNN) algorithms are shown to be the most robust. Namely, these algorithms can detect the presence of heart murmurs even without a heartbeat alignment step. Furthermore, Support Vector Machine (SVM) and Random Forest (RF) algorithms require an explicit segmentation step to effectively detect heart sounds and murmurs, the overall performance is expected drop approximately 5% on both cases.

Line Art Colorization Based on Explicit Region Segmentation

AbstractAutomatic line art colorization plays an important role in anime and comic industry. While existing methods for line art colorization are able to generate plausible colorized results, they tend to suffer from the color bleeding issue. We introduce an explicit segmentation fusion mechanism to aid colorization frameworks in avoiding color bleeding artifacts. This mechanism is able to provide region segmentation information for the colorization process explicitly so that the colorization model can learn to avoid assigning the same color across regions with different semantics or inconsistent colors inside an individual region. The proposed mechanism is designed in a plug‐and‐play manner, so it can be applied to a diversity of line art colorization frameworks with various kinds of user guidances. We evaluate this mechanism in tag‐based and reference‐based line art colorization tasks by incorporating it into the state‐of‐the‐art models. Comparisons with these existing models corroborate the effectiveness of our method which largely alleviates the color bleeding artifacts. The code is available athttps://github.com/Ricardo-L-C/ColorizationWithRegion.

Dynamic to Static Lidar Scan Reconstruction Using Adversarially Trained Auto Encoder

Accurate reconstruction of static environments from LiDAR scans of scenes containing dynamic objects, which we refer to as Dynamic to Static Translation (DST), is an important area of research in Autonomous Navigation. This problem has been recently explored for visual SLAM, but to the best of our knowledge no work has been attempted to address DST for LiDAR scans. The problem is of critical importance due to wide-spread adoption of LiDAR in Autonomous Vehicles. We show that state-of the art methods developed for the visual domain when adapted for LiDAR scans perform poorly. We develop DSLR, a deep generative model which learns a mapping between dynamic scan to its static counterpart through an adversarially trained autoencoder. Our model yields the first solution for DST on LiDAR that generates static scans without using explicit segmentation labels. DSLR cannot always be applied to real world data due to lack of paired dynamic-static scans. Using Unsupervised Domain Adaptation, we propose DSLR-UDA for transfer to real world data and experimentally show that this performs well in real world settings. Additionally, if segmentation information is available, we extend DSLR to DSLR-Seg to further improve the reconstruction quality. DSLR gives the state of the art performance on simulated and real-world datasets and also shows at least 4× improvement. We show that DSLR, unlike the existing baselines, is a practically viable model with its reconstruction quality within the tolerable limits for tasks pertaining to autonomous navigation like SLAM in dynamic environments.

Short duration Vectorcardiogram based inferior myocardial infarction detection: class and subject-oriented approach.

Myocardial infarction (MI) happens when blood stops circulating to an explicit segment of the heart causing harm to the heart muscles. Vectorcardiography (VCG) is a technique of recording direction and magnitude of the signals that are produced by the heart in a 3-lead representation. In this work, we present a technique for detection of MI in the inferior portion of heart using short duration VCG signals. The raw signal was pre-processed using the median and Savitzky-Golay (SG) filter. The Stationary Wavelet Transform (SWT) was used for time-invariant decomposition of the signal followed by feature extraction. The selected features using minimum-redundancy-maximum-relevance (mRMR) based feature selection method were applied to the supervised classification methods. The efficacy of the proposed method was assessed under both class-oriented and a more real-life subject-oriented approach. An accuracy of 99.14 and 89.37% were achieved respectively. Results of the proposed technique are better than existing state-of-art methods and used VCG segment is shorter. Thus, a shorter segment and a high accuracy can be helpful in the automation of timely and reliable detection of MI. The satisfactory performance achieved in the subject-oriented approach shows reliability and applicability of the proposed technique.

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.