Frame Segmentation Research Articles

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions.

Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears). With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames. We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially. Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.

Crane payload localisation for curtain wall installation: A markerless computer vision approach

Automated measurement of the relative pose between a crane borne curtain wall module and its installation location on the side face of a high-rise building can be applied to increase the safety and efficiency of crane operations though informing the action required to achieve alignment. However, the detection and measurement tasks are challenging because the construction site is large, unstructured, and highly dynamic. This article introduces a markerless computer vision measurement algorithm and a practical implementation, which uses a forward-facing infrared camera attached to the crane spreader. The algorithm self-verifies the measurement against known information so that it can fail safely instead of returning a malformed measurement. The algorithm is experimentally validated in challenging lighting conditions. The window frame segmentation achieved Fβ=0.59. Overall, the algorithm returned 71% successful and 0 malformed measurements.

A Four-Point Camera Calibration Method for Sport Videos

The main objective of camera calibration is to estimate the homography between a sports field template and the corresponding field in a video frame. State-of-the-art approaches exploit deep-learning technologies to achieve this aim. However, the lack of large training datasets leads to unreliable results. Search-based approaches deliver better accuracy but at the expense of a very high computational cost. In this paper, a four-point camera calibration method is proposed to address these challenges. It consists of four stages: first, a conditional generative adversarial network (cGAN) is used to generate meaningfully segmented video frames. This cGAN is a general image-to-image translation technique which has robust performance even for small training datasets. The aim is to eliminate foreground objects that greatly impede accurate estimation of four suitable points for the subsequent stage. Then, a regression network which can estimate four points from a single segmented video frame is introduced. In this stage we aim at keeping the low computational cost. In the next processing stage, the estimated four points and the original selected four points are used to calculate the homography. In this third stage a Direct Linear Transformation (DLT) algorithm is applied. In the last processing stage, the accuracy of the estimated homography is refined by running an Enhanced Correlation Coefficient (ECC) module. The proposed method is extensively evaluated using a standard 2014 World Cup dataset. The obtained results are compared to the state-of-the-art approaches and its variations based on U-net, VGG-16, ResNet-50. The obtained results show that the proposed method is superior in terms of accuracy and computational efficiency. To demonstrate the effectiveness and robustness of the proposed method, it is also validated on National Basketball, 2018 World Cup and cross dataset. In these diverse environments, the proposed method still maintains the competitive performance.

Complementary Coarse-to-Fine Matching for Video Object Segmentation

Semi-supervised Video Object Segmentation (VOS) needs to establish pixel-level correspondences between a video frame and preceding segmented frames to leverage their segmentation clues. Most works rely on features at a single scale to establish those correspondences, e.g., perform dense matching with Convolutional Neural Network (CNN) features from a deep layer. Differently, this work explores complementary features at different scales to pursue more robust feature matching. A coarse feature from a deep layer is first adopted to get coarse pixel-level correspondences. We hence evaluate the quality of those correspondences, and select pixels with low-quality correspondences for fine-scale feature matching. Segmentation clues of previous frames are propagated by both coarse and fine-scale correspondences, which are fused with appearance features for object segmentation. Compared with previous works, this coarse-to-fine matching scheme is more robust to distractions by similar objects and better preserves object details. The sparse fine-scale matching also ensures a fast inference speed. On popular VOS datasets including DAVIS and YouTube-VOS, the proposed method shows promising performance compared with recent works.

Fully automatic tracking of native glenohumeral kinematics from stereo-radiography

The current work introduces a system for fully automatic tracking of native glenohumeral kinematics in stereo-radiography sequences. The proposed method first applies convolutional neural networks to obtain segmentation and semantic key point predictions in biplanar radiograph frames. Preliminary bone pose estimates are computed by solving a non-convex optimization problem with semidefinite relaxations to register digitized bone landmarks to semantic key points. Initial poses are then refined by registering computed tomography-based digitally reconstructed radiographs to captured scenes, which are masked by segmentation maps to isolate the shoulder joint. A particular neural net architecture which exploits subject-specific geometry is also introduced to improve segmentation predictions and increase robustness of subsequent pose estimates. The method is evaluated by comparing predicted glenohumeral kinematics to manually tracked values from 17 trials capturing 4 dynamic activities. Median orientation differences between predicted and ground truth poses were 1.7∘ and 8.6∘ for the scapula and humerus, respectively. Joint-level kinematics differences were less than 2∘ in 65%, 13%, and 63% of frames for XYZ orientation DoFs based on Euler angle decompositions. Automation of kinematic tracking can increase scalability of tracking workflows in research, clinical, or surgical applications.

Defect detection and response non-uniformity correction of a monocentric camera based on fiber optic relay imaging.

The monocentric camera based on fiber relay imaging offers benefits of light weight, compact size envelope, vast field of view, and high resolution, which can fully fulfill the index requirements of space-based surveillance systems. However, the fiber optic plate's (FOP) defects will result in the loss of imaging data, and the FOP's discrete structural features will exacerbate the imaging's non-uniformity. A global defect detection approach based on manual threshold segmentation of saturated frames is suggested to detect FOP defect features. The suggested method's efficacy and accuracy are confirmed when compared to the classical Otsu algorithm. Additionally, through tests, the relative imaging response coefficients of each pixel are identified, the response non-uniformity of the pixels is corrected, and the whole image non-uniformity drops from 10.01% to 0.78%. The study in this paper expedites the use of fiber relay imaging-based monocentric cameras in the field of space-based surveillance, and the technique described in this paper is also appropriate for large-array optical fiber coupled relay image transmission systems.

Eye-Blink Event Detection Using a Neural-Network-Trained Frame Segment for Woman Drivers in Saudi Arabia

Women have been allowed to drive in Saudi Arabia since 2018, revoking a 30-year ban that also adhered to the traffic rules provided in the country. Conventional drivers are often monitored for safe driving by monitoring their facial reactions, eye blinks, and expressions. As driving experience and vehicle handling features have been less exposed to novice women drivers in Saudi Arabia, technical assistance and physical observations are mandatory. Such observations are sensed as images/video frames for computer-based analyses. Precise computer vision processes are employed for detecting and classifying events using image processing. The identified events are unique to novice women drivers in Saudi Arabia, assisting with their vehicle usage. This article introduces the Event Detection using Segmented Frame (ED-SF) method to improve the abnormal Eye-Blink Detection (EBD) of women drivers. The eye region is segmented using variation pixel extraction in this process. The pixel extraction process requires textural variation identified from different frames. The condition is that the frames are to be continuous in the event detection. This method employs a convolution neural network with two hidden layer processes. In the first layer, continuous and discrete frame differentiations are identified. The second layer is responsible for segmenting the eye region, devouring the textural variation. The variations and discrete frames are used for training the neural network to prevent segment errors in the extraction process. Therefore, the frame segment changes are used for Identifying the expressions through different inputs across different texture luminosities. This method applies to less-experienced and road-safety-knowledge-lacking woman drivers who have initiated their driving journey in Saudi-Arabia-like countries. Thus the proposed method improves the EBD accuracy by 9.5% compared to Hybrid Convolutional Neural Networks (HCNN), Long Short-Term Neural Networks (HCNN + LSTM), Two-Stream Spatial-Temporal Graph Convolutional Networks (2S-STGCN), and the Customized Driving Fatigue Detection Method CDFDM.

A frame orientation optimisation method for consistent interpretation of kinematic signals

In clinical movement biomechanics, kinematic data are often depicted as waveforms (i.e. signals), characterising the motion of articulating joints. Clinically meaningful interpretations of the underlying joint kinematics, however, require an objective understanding of whether two different kinematic signals actually represent two different underlying physical movement patterns of the joint or not. Previously, the accuracy of IMU-based knee joint angles was assessed using a six-degrees-of-freedom joint simulator guided by fluoroscopy-based signals. Despite implementation of sensor-to-segment corrections, observed errors were clearly indicative of cross-talk, and thus inconsistent reference frame orientations. Here, we address these limitations by exploring how minimisation of dedicated cost functions can harmonise differences in frame orientations, ultimately facilitating consistent interpretation of articulating joint kinematic signals. In this study, we present and investigate a frame orientation optimisation method (FOOM) that aligns reference frames and corrects for cross-talk errors, hence yielding a consistent interpretation of the underlying movement patterns. By executing optimised rotational sequences, thus producing angular corrections around each axis, we enable a reproducible frame definition and hence an approach for reliable comparison of kinematic data. Using this approach, root-mean-square errors between the previously collected (1) IMU-based data using functional joint axes, and (2) simulated fluoroscopy-based data relying on geometrical axes were almost entirely eliminated from an initial range of 0.7°–5.1° to a mere 0.1°–0.8°. Our results confirm that different local segment frames can yield different kinematic patterns, despite following the same rotation convention, and that appropriate alignment of reference frame orientation can successfully enable consistent kinematic interpretation.

Abnormal event detection model using an improved ResNet101 in context aware surveillance system

AbstractSurveillance system plays a significant role for achieving security monitoring in the place of crowd areas. Offline monitoring of these crowd activity is quite challenging because it requires huge number of human resources for attaining efficient tracking. For shortcoming these issue automated and intelligent based system must be developed for efficiently monitor crowd and detect abnormal activity. However the existing methods faces issues like irrelevant features, high cost and process complexity. In this current research context aware surveillance‐system utilising hybrid ResNet101‐ANN is developed for effective abnormal activity detection. For this proposed approach video acquired from surveillance camera is considered as input. Then, acquired video is segmented into multiple frames. After that pre‐processing techniques such as denoising using mean filter, motion deblurring, contrast enhancement using Histogram Equalisation and canny edge detection is applied in this segmented frames. Further, the pre‐processed frame is fetched into hybrid ResNet101‐ANN classifier for abnormal event classification. Here, ResNet101 is used for extracting the features from the frames and Artificial neural network which replaces the fully connected layer of ResNet101 us used to detect the abnormal activity. If once abnormal‐events detected the context aware services generate alert to the user for preventing abnormal‐activities. Accuracy, precision, recall, and error values reached for the proposed‐model on simulation were 0.98, 0.98, 0.98 and 0.017 respectively. Using this proposed model effective crowd monitoring and abnormal activity detection can be achieved.

Attribution and the discourse structure of reports

I propose a discourse-level analysis of report constructions. Indirect discourse, mixed and direct quotation, free indirect discourse, and attitude ascriptions are all analyzed in terms of a discourse relation of ATTRIBUTION, connecting two propositional discourse units corresponding to (i) a frame segment (he said, she dreamed) and a (possibly complex, multi-sentence) report (“I’m an idiot”, (that) she was president). I provide a unified semantics for the discourse relation of ATTRIBUTION that invokes a flexible notion of ‘characterization’. A discourse unit may characterize a speech event by reproducing its linguistic surface form (as in quotation) or its propositional content (as in indirect speech and attitude reports), or some mixture of both (as in mixed quotation or free indirect discourse). I formalize this unified discourse-level ATTRIBUTION approach to reporting within the general framework of SDRT, and apply it to direct, indirect, and free indirect reports that extend beyond the single embedded or quoted clause. The resulting account is the first to do justice to the complex internal dependencies within stretches of reported discourse.

Global video object segmentation with spatial constraint module

We present a lightweight and efficient semi-supervised video object segmentation network based on the space-time memory framework. To some extent, our method solves the two difficulties encountered in traditional video object segmentation: one is that the single frame calculation time is too long, and the other is that the current frame’s segmentation should use more information from past frames. The algorithm uses a global context (GC) module to achieve high-performance, real-time segmentation. The GC module can effectively integrate multi-frame image information without increased memory and can process each frame in real time. Moreover, the prediction mask of the previous frame is helpful for the segmentation of the current frame, so we input it into a spatial constraint module (SCM), which constrains the areas of segments in the current frame. The SCM effectively alleviates mismatching of similar targets yet consumes few additional resources. We added a refinement module to the decoder to improve boundary segmentation. Our model achieves state-of-the-art results on various datasets, scoring 80.1% on YouTube-VOS 2018 and a {cal J}{rm{& }}{cal F} score of 78.0% on DAVIS 2017, while taking 0.05 s per frame on the DAVIS 2016 validation dataset.

Enhanced security using video summarization for surveillance system using deep LSTM model with K-means clustering technique

For the security of the Society, Video surveillance systems have become an important tool that can help in preventing crime in public or private establishments, leading to more safe and secure environment. With the rise of technology that can handle high volune video data and the need for continuous monitoring for security to citizens, the analysis of the video footages, during security breach attempts has become more challenging. Video summarization can play a crucial role in secure surveillance systems by optimizing the analysis time of the security breach. This research attempts to summarise the important aspects of the original video without losing its context and also optimizes the storage requirement. To achieve the same a unique KM-LSTM based technique has been developed that utilizes deep learning methods to understand links among highlight as well as non-highlight video segments through the use of pairwise distance calculations. A new Deep LSTM model has been used to extract the features from all segments of frames. These extracted features are clustered using k-means clustering. Then, such features are trained using a pairwise deep learning model to get highlight scores as well as leverage the comparative similarity among pairs of the highlight as well as non-highlight segments. At last, video summarization takes place to generate the video summary from the highlight segments only. From the results of these experiments discovering the moments of the user’s major or special interest (namely, highlights) in a video, to generating summarization of videos. The above technique can help in increasing the analysis of Video content and thus achieving a more vigilant and secure society.

Features of forming process of frame segments blanks based on stretch bending technology of extruded section made of high-strength aluminum alloys

The methods for manufacturing of the frames of the aircraft are analyzed. The most optimal technological route for the manufacture of frames is selected. The process of stretch bending is considered, the problems arising during its implementation are shown. The ways for improving of the stretch bending process described in open sources are analyzed. The parameters necessary for the development of an adaptive control system for the stretch bending process are determined.

An Efficient Attention-Based Strategy for Anomaly Detection in Surveillance Video

In the present technological world, surveillance cameras generate an immense amount of video data from various sources, making its scrutiny tough for computer vision specialists. It is difficult to search for anomalous events manually in these massive video records since they happen infrequently and with a low probability in real-world monitoring systems. Therefore, intelligent surveillance is a requirement of the modern day, as it enables the automatic identification of normal and aberrant behavior using artificial intelligence and computer vision technologies. In this article, we introduce an efficient Attention-based deep-learning approach for anomaly detection in surveillance video (ADSV). At the input of the ADSV, a shots boundary detection technique is used to segment prominent frames. Next, The Light-weight Convolution Neural Network (LWCNN) model receives the segmented frames to extract spatial and temporal information from the intermediate layer. Following that, spatial and temporal features are learned using Long Short-Term Memory (LSTM) cells and Attention Network from a series of frames for each anomalous activity in a sample. To detect motion and action, the LWCNN received chronologically sorted frames. Finally, the anomaly activity in the video is identified using the proposed trained ADSV model. Extensive experiments are conducted on complex and challenging benchmark datasets. In addition, the experimental results have been compared to state-of-the-art methodologies, and a significant improvement is attained, demonstrating the efficiency of our ADSV method.

LED Screen-Based Intelligent Hand Gesture Recognition System

Common human–computer interaction (HCI) modes rely on various sensors or cameras for operation. This study proposes a gesture recognition method and device based on light sensing characteristics, which removes the need for external sensors with only light-emitting diode (LED) screens. The key technique is that the photoelectric sensing ability of LED is applied to detect the information change of light without interference to the display function, thus indicating the variation of gestures. This system is integrated by using six LED screen modules with the field-programmable gate array (FPGA) to control display and collect data for deep-learning analysis. Depending on both time- and frequency-domain characteristics of gestures, the recognition algorithm is implemented based on static bidirectional long short-term memory (S-Bi-LSTM). As revealed by the experiments conducted on hand gestures of different people and under the context of complex background illumination, the accuracy of gesture recognition reaches as high as 93.60%. As for dynamic gesture actions, an optimized dynamic bidirectional long short-term memory (D-Bi-LSTM) algorithm is also proposed with the method of frame segmentation. This method divides dynamic gestures into three parts to feed different classifiers, so as to enhance the accuracy of dynamic gestures recognition up to 91.67%. The device possesses all the structures of a large LED screen, which verifies the feasibility of pattern recognition for large LED screens from an experimental perspective. Without any need for traditional sensors, it is contactless, secure, and easy to promote.

Coherence-aware context aggregator for fast video object segmentation

Semi-supervised video object segmentation (VOS) is a highly challenging problem that has attracted much research attention in recent years. Temporal context plays an important role in VOS by providing object clues from the past frames. However, most of the prevailing methods directly use the predicted temporal results to guide the segmentation of the current frame, while ignoring the coherence of temporal context, which may be misleading and degrade the performance. In this paper, we propose a novel model named Coherence-aware Context Aggregator (CCA) for VOS, which consists of three modules. First, a coherence-aware module (CAM) is proposed to evaluate the coherence of the predicted result of the current frame and then fuses the coherent features to update the temporal context. CAM can determine whether the prediction is accurate, thus guiding the update of the temporal context and avoiding the introduction of erroneous information. Second, we devise a spatio-temporal context aggregation (STCA) module to aggregate the temporal context with the spatial feature of the current frame to learn a robust and discriminative target representation in the decoder part. Third, we design a refinement module to refine the coarse feature generated from the STCA module for more precise segmentation. Additionally, CCA uses a cropping strategy and takes small-size images as input, thus making it computationally efficient and achieving a real-time running speed. Extensive experiments on four challenging benchmarks show that CCA achieves a better trade-off between efficiency and accuracy compared to state-of-the-art methods. The code will be public.

Classification of Valvular Regurgitation Using Echocardiography

Echocardiography (echo) is a commonly utilized tool in the diagnosis of various forms of valvular heart disease for its ability to detect types of cardiac regurgitation. Regurgitation represents irregularities in cardiac function and the early detection of regurgitation is necessary to avoid invasive cardiovascular surgery. In this paper, we focussed on the classification of regurgitations from videographic echo images. Three different types of regurgitation are considered in this work, namely, aortic regurgitation (AR), mitral regurgitation (MR), and tricuspid regurgitation (TR). From the echo images, texture features are extracted, and classification is performed using Random Forest (RF) classifier. Extraction of keyframe is performed from the video file using two approaches: a reference frame keyframe extraction technique and a redundant frame removal technique. To check the robustness of the model, we have considered both segmented and nonsegmented frames. Segmentation is carried out after keyframe extraction using the Level Set (LS) with Fuzzy C-means (FCM) approach. Performances are evaluated in terms of accuracy, precision, recall, and F1-score and compared for both reference frame and redundant frame extraction techniques. K-fold cross-validation is used to examine the performance of the model. The performance result shows that our proposed approach outperforms other state-of-art machine learning approaches in terms of accuracy, precision, recall, and F1-score.

A coarse-to-fine segmentation frame for polyp segmentation via deep and classification features

Accurate polyp segmentation is of great significance for the diagnosis and treatment of colon cancer. Deep convolution network can extract the common high level features of the target. However, most network models ignore some individual features, so the sample prediction results in complex space are fuzzy and lack of regularity. In this paper, a coarse-to-fine segmentation frame for polyp segmentation via deep and classification features is proposed. Firstly, batch schatten-p norms maximization is introduced into a network model to strengthen the predict map. Then, an automatic two classification mechanism is constructed and the prediction map is classified into two categories: simple and complex samples. Since the CNN prediction maps of simple samples are close to binary images, the prediction maps are not processed. Finally, an active contour model segmentation algorithm for saliency detection of complex samples is proposed. Experiments on Kvasir-SEG, CVC-300, CVC-ClinincDB, CVC-ColonDB and ETIS-LaribPolypDB datasets using multiple models verify the effectiveness of the framework. Code is available at https://doi.org/10.24433/CO.7821162.v1.

Saying the Unseen: Video Descriptions via Dialog Agents.

Current vision and language tasks usually take complete visual data (e.g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e.g., restricted view with fixed camera or intentional vision block for security concerns. As a step towards the more practical application scenarios, we introduce a novel task that aims to describe a video using the natural language dialog between two agents as a supplementary information source given incomplete visual data. Different from most existing vision-language tasks where AI systems have full access to images or video clips, which may reveal sensitive information such as recognizable human faces or voices, we intentionally limit the visual input for AI systems and seek a more secure and transparent information medium, i.e., the natural language dialog, to supplement the missing visual information. Specifically, one of the intelligent agents - Q-BOT - is given two semantic segmented frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has access to the entire video, assists Q-BOT to accomplish the goal by answering the asked questions. We introduce two different experimental settings with either a generative (i.e., agents generate questions and answers freely) or a discriminative (i.e., agents select the questions and answers from candidates) internal dialog generation process. With the proposed unified QA-Cooperative networks, we experimentally demonstrate the knowledge transfer process between the two dialog agents and the effectiveness of using the natural language dialog as a supplement for incomplete implicit visions.

A Generic Approach towards Amharic Sign Language Recognition

In the day-to-day life of communities, good communication channels are crucial for mutual understanding. The hearing-impaired community uses sign language, which is a visual and gestural language. In terms of orientation and expression, it is separate from written and spoken languages. Despite the fact that sign language is an excellent platform for communication among hearing-impaired persons, it has created a communication barrier between hearing-impaired and non-disabled people. To address this issue, researchers have proposed sign language to text translation systems for English and other European languages as a solution. The goal of this research is to design and develop an Amharic digital text converter system using Ethiopian sign language. The proposed system was created with the help of two key deep learning algorithms: a pretrained deep learning model and a Long Short-Term Memory (LSTM). The LSTM was used to extract sequence information from a sequence of image frames of a specific sign language, while the pretrained deep learning model was used to extract features from single frame images. The dataset used to train the algorithms was gathered in video format from Addis Ababa University. Prior to feeding the obtained dataset to the deep learning models, data preprocessing activities such as cleaning and video to image frame segmentation were conducted. The system was trained, validated, and tested using 80%, 10%, and 10% of the 2475 images created during the preprocessing step. Two pretrained deep learning models, EfficientNetB0 and ResNet50, were used in this investigation, and they attained an accuracy of 72.79%. In terms of precision and f1-score, ResNet50 outperformed EfficientNetB0. For the proposed system, a graphical user interface prototype was created, and the best performing model was chosen and implemented. The proposed system can be utilized as a starting point for other researchers to improve upon, based on the outcomes of the experiment. More high-quality training datasets and high-performance training machines, such as GPU-enabled computers, can be added to the system to improve it.