Optical Flow Research Articles

Development of a body movement detection system to avoid re-exposure during radiography.

During the radiographic examination of the chest and bones in hospitals, communicating and maintaining posture is difficult for some patients, and movement before or during X-ray irradiation may necessitate re-exposure owing to body wobbling movements or breathing movements. To prevent the need for re-exposure during radiography and to determine the exposure timing, a body movement detection system that considers breathing movements was developed in this study. The posture of a patient was monitored using an RGB camera. The acquired video data was analyzed to detect body movement using either an inter-frame difference method or an optical flow estimation method. The performance of the system was evaluated by detecting the body and breathing movements during positioning. Consequently, the inter-frame difference method detected 179.8-1222.2pixels during body movements, and the optical flow estimation method confirmed that the feature points moved by 5.5-26.6mm (4.2-20.3pixels). When detecting breathing movements, 82-585pixels were detected by the inter-frame difference method, and the optical flow estimation method showed that the feature points moved by 5.2mm (2-4pixels). Therefore, the proposed method can detect body movements during radiography to prevent re-exposure due to body wobble and breathing movements. For healthcare providers, it will lead to reduce not only concerns about patient exposure but also unnecessary radiographic workload.

Value of Combined Optical Coherence Tomography and Optical Flow Ratio Measurements After Percutaneous Coronary Intervention

BackgroundOptical flow ratio (OFR) is a novel computational fractional flow reserve derived from optical coherence tomography (OCT). However, the impact of combining post-stenting morphology (OCT) and physiology (OFR) remains largely unknown. MethodsOCT and OFR were analysed at an independent core laboratory. Target lesion failure (TLF) was defined as the composite of cardiac death, target lesion myocardial infarction, and target lesion revascularisation. Suboptimal stent deployment was identified with at least 1 TLF-related OCT or OFR characteristic. ResultsA total of 448 patients with acute coronary syndrome (459 vessels) were assessed. Stent expansion < 80%, minimal stent area < 4.5 mm2, stent edge lipid-rich plaque and OFR < 0.90 were independent predictors of TLF (all P < 0.001). Patients with OCT-suboptimal (adjusted hazard ratio [aHR] 7.88, 95% CI 2.73-22.72,-P < 0.001) or OFR-suboptimal (aHR 5.78, 95% CI 2.54-13.14; P < 0.001) stent deployment showed significantly higher risk of TLF compared with those with optimal stent deployment, with a significant interaction (Pinteraction < 0.001). OCT and OFR both–suboptimal stent deployment was confirmed as an independent predictor of TLF (aHR 9.39, 95% CI 4.25-20.76; P < 0.001). ConclusionsCombined OCT and OFR conferred an optimal reclassification of stent deployment, which may aid in decision making regarding a tailored PCI strategy for optimal stent deployment.

Consistency-constrained unsupervised video anomaly detection framework based on Co-teaching

Intelligent video surveillance continues to be a vibrant research domain within the field of computer vision. However, existing representation learning frameworks primarily focus on static information extraction frame by frame such as appearance features, they often overlook the valuable dynamic information like optical flow feature inherent in the video data, which is most essential characteristics of sequence data. To mining dynamic features and bridge this gap, our paper introduces a novel anomaly detection framework that balance dynamic information with static information and construct a relationship between appearance features and corresponding optical flow features, where we sets strong consistency constraints, which reduce the loss between dynamic information and corresponding static information, and we leverages collaborative teaching network to ensure a consistent representation of both static and dynamic information for predict. The proposed framework consists of two sets of encoder–decoder pairs complemented by a memory storage module. Operating in parallel with the dual encoder network is a Co-teaching network, with the shared memory module serving as the cornerstone for collaborative training. The Consistency constrained condition guarantees the strong consistency of dynamic and static information in the learned representations. In our experimental phase, we present compelling results that showcase the superior performance of our algorithm across three publicly available datasets.

Optical Flow Based Detection and Tracking of Moving Objects for Autonomous Vehicles

Optical Flow Based Detection and Tracking of Moving Objects for Autonomous Vehicles

Development of an Automatic Feature Point Classification Method for Three-Dimensional Mapping Around Slewing and Derricking Cranes

Crane automation requires a three-dimensional (3D) map around cranes that should be reconstructed and updated quickly.In this study, a high-precision classification method was developed to distinguish stationary objects from moving objects in moving images captured by a monocular camera to stabilize 3D reconstruction. To develop the method, a moving image was captured while the crane was slewed with a monocular camera mounted vertically downward at the tip of the crane. The boom length and angle data were output from a control device, a controller area network. For efficient development, a simulator that imitated the environment of an actual machine was developed and used. The proposed method uses optical flow to track feature points. The classification was performed successfully, independent of derricking motion. Consequently, the proposed method contributes to stable 3D mapping around cranes in construction sites.

Reviewing Approaches and Techniques for Detecting Suspicious Human Behavior: A Comprehensive Survey

The paper aims to review related works that focus on detecting suspicious human behavior using machine-learning techniques. Suspicious human behavior refers to behaviors that may indicate involvement in or preparation for a crime. Detecting such behaviors before a crime is committed allows law enforcement to take early action and prevent criminal activities. One of the challenges in developing an effective detection system for suspicious human behavior is the absence of a well-defined definition for such behaviors. Different definitions can lead to various methods for designing the detection system. The paper explores different definitions and their implications on the design of detection systems and mentions two types of methods that can be used for detecting suspicious human behavior: image-based methods and saliency mapping. Image-based methods utilize image or video recognition techniques to analyze objects held by individuals or recognize specific activities. Saliency mapping, on the other hand, focuses on emphasizing the movement of individuals using techniques like optical flow calculation to generate saliency maps. Additionally, the paper highlights the increasing popularity of embedded machine learning, particularly on portable platforms. The use of embedded machine learning allows for the deployment of machine learning models on mobile or lightweight devices. This can be relevant for developing efficient and portable systems for detecting suspicious human behavior. Overall, the paper aims to provide an overview of existing works in the field of suspicious human behavior detection using machine learning, exploring different definitions and methods employed in the literature.

Genome-wide analysis of the biophysical properties of chromatin and nuclear proteins in living cells with Hi-D.

To understand the dynamic nature of the genome, the localization and rearrangement of DNA and DNA-binding proteins must be analyzed across the entire nucleus of single living cells. Recently, we developed a computational light microscopy technique, called high-resolution diffusion (Hi-D) mapping, which can accurately detect, classify and map diffusion dynamics and biophysical parameters such as the diffusion constant, the anomalous exponent, drift velocity and model physical diffusion from the data at a high spatial resolution across the genome in living cells. Hi-D combines dense optical flow to detect and track local chromatin and nuclear protein motion genome-wide and Bayesian inference to characterize this local movement at nanoscale resolution. Here we present the Python implementation of Hi-D, with an option for parallelizing the calculations to run on multicore central processing units (CPUs). The functionality of Hi-D is presented to the users via user-friendly documented Python notebooks. Hi-D reduces the analysis time to less than 1 h using a multicore CPU with a single compute node. We also present different applications of Hi-D for live-imaging of DNA, histone H2B and RNA polymerase II sequences acquired with spinning disk confocal and super-resolution structured illumination microscopy.

AI-Based Integrated Smart Process Sensor for Emulsion Control in Industrial Application

In industry, reliable process supervision is essential to ensure efficient, safe, and high-quality production. The droplet size distribution represents a critical quality attribute for emulsification processes and should be monitored. For emulsion characterization, image-based analysis methods are well-known but are often performed offline, leading to a time-delayed and error-prone process evaluation. The use of an integrated smart process sensor to characterize the emulsification process over time enables the real-time evaluation of the entire system. The presented integrated smart process sensor consists of an optical measurement flow cell built into a camera system. The overall system is placed in a bypass system of a production plant for emulsification processes. AI-based image evaluation is used in combination with a feature extraction method (You Only Look Once version 4 (YOLOv4) and Hough circle (HC)) to characterize the process over time. The sensor system is installed in the plant and tested with different cosmetic products. Various iteration, prototyping, and test steps for the final sensor design are performed prior to this in a laboratory test setup. The results indicate robust and accurate detection and determination of the droplet size in real time to improve product control and save time. For benchmarking the integrated smart process sensor, the results are compared with common analysis methods using offline samples.

Ghost cells as a two-phase blood analog fluid-high-volume and high-concentration production.

Hemolysis in mechanical circulatory support systems is currently determined quantitatively. To also locally resolve hemolysis, we are developing a fluorescent hemolysis detection method. This requires a translucent two-phase blood analog fluid combined with particle image velocimetry, an optical flow field measurement. The blood analog fluid is composed of red blood cell surrogates. However, producing surrogates in sufficient volume is a challenge. We therefore present a high-volume and high-concentration production for our surrogates: ghost cells, hemoglobin-depleted erythrocytes. In the ghost cell production, the hemoglobin is removed by a repeated controlled osmolar lysis. We have varied the solution mixture, centrifugation time, and centrifugation force in order to increase production efficiency. The production is characterized by measurements of output volume, hematocrit, transparency, and rheology of the blood analog fluid. The volume of produced ghost cells was significantly increased, and reproducibility was improved. An average production of 389 mL of ghost cells were achieved per day. Those ghost cells diluted in plasma have a rheology similar to blood while being permeable to light. The volume of ghost cells produced is sufficient for optical measurements as particle image velocimetry in mechanical circulatory support systems. This makes further work on experimental measurements for a locally resolved hemolysis detection possible.

Crowd behavior detection: leveraging video swin transformer for crowd size and violence level analysis

In recent years, crowd behavior detection has posed significant challenges in the realm of public safety and security, even with the advancements in surveillance technologies. The ability to perform real-time surveillance and accurately identify crowd behavior by considering factors such as crowd size and violence levels can avert potential crowd-related disasters and hazards to a considerable extent. However, most existing approaches are not viable to deal with the complexities of crowd dynamics and fail to distinguish different violence levels within crowds. Moreover, the prevailing approach to crowd behavior recognition, which solely relies on the analysis of closed-circuit television (CCTV) footage and overlooks the integration of online social media video content, leads to a primarily reactive methodology. This paper proposes a crowd behavior detection framework based on the swin transformer architecture, which leverages crowd counting maps and optical flow maps to detect crowd behavior across various sizes and violence levels. To support this framework, we created a dataset comprising videos capable of recognizing crowd behaviors based on size and violence levels sourced from CCTV camera footage and online videos. Experimental analysis conducted on benchmark datasets and our proposed dataset substantiates the superiority of our proposed approach over existing state-of-the-art methods, showcasing its ability to effectively distinguish crowd behaviors concerning size and violence level. Our method’s validation through Nvidia’s DeepStream Software Development Kit (SDK) highlights its competitive performance and potential for real-time intelligent surveillance applications.Graphical abstract

Improved high dynamic range imaging using multi-scale feature flows balanced between task-orientedness and accuracy

Deep learning has made it possible to accurately generate high dynamic range (HDR) images from multiple images taken at different exposure settings, largely owing to advancements in neural network design. However, generating images without artifacts remains difficult, especially in scenes with moving objects. In such cases, issues like color distortion, geometric misalignment, or ghosting can appear. Current state-of-the-art network designs address this by estimating the optical flow between input images to align them better. The parameters for the flow estimation are learned through the primary goal, producing high-quality HDR images. However, we find that this ”task-oriented flow” approach has its drawbacks, especially in minimizing artifacts. To address this, we introduce a new network design and training method that improve the accuracy of flow estimation. This aims to strike a balance between task-oriented flow and accurate flow. Additionally, the network utilizes multi-scale features extracted from the input images for both flow estimation and HDR image reconstruction. Our experiments demonstrate that these two innovations result in HDR images with fewer artifacts and enhanced quality.

An unsupervised video anomaly detection method via Optical Flow decomposition and Spatio-Temporal feature learning

The purpose of this paper is to present an unsupervised video anomaly detection method using Optical Flow decomposition and Spatio-Temporal feature learning (OFST). This method employs a combination of optical flow reconstruction and video frame prediction to achieve satisfactory results. The proposed OFST framework is composed of two modules: the Multi-Granularity Memory-augmented Autoencoder with Optical Flow Decomposition (MG-MemAE-OFD) and a Two-Stream Network based on Spatio-Temporal feature learning (TSN-ST). The MG-MemAE-OFD module is composed of three functional blocks: optical flow decomposition, autoencoder, and multi-granularity memory networks. The optical flow decomposition block is used to extract the main motion information of objects in optical flow, and the granularity memory network is utilized to memorize normal patterns and improve the quality of the reconstructions. To predict video frames, we introduce a two-stream network based on spatiotemporal feature learning (TSN-ST), which adopts parallel standard Transformer blocks and a temporal block to learn spatiotemporal features from video frames and optical flows. The OFST combines these two modules so that the prediction error of abnormal samples is further increased due to the larger reconstruction error. In contrast, the normal samples obtain a lower reconstruction error and prediction error. Therefore, the anomaly detection capability of the method is greatly enhanced. Our proposed model was evaluated on public datasets. Specifically, in terms of the area under the curve (AUC), our model achieved an accuracy of 85.74% on the Ped1 dataset, 99.62% on the Ped2 dataset, 93.89% on the Avenue dataset, and 76.0% on the ShanghaiTech Dataset. Our experimental results show an average improvement of 1.2% compared to the current state-of-the-art.

A novel 3D reconstruction method of blast furnace burden surface based on virtual camera array

Accurate three-dimensional (3D) blast furnace burden surface topography is crucial for optimizing material distribution and furnace operation. However, harsh conditions within the furnace pose challenges to this 3D reconstruction, such as difficulties in matching feature points and sparse 3D point clouds. To overcome these challenges, we propose a 3D reconstruction method for the blast furnace burden surface using virtual camera array. Firstly, an ORB feature extraction and matching method based on bidirectional optical flow tracking is proposed to solve the impact of illumination changes and noise. Then, the virtual camera array is constructed to generate a sparse 3D point cloud from an image sequence. Finally, dense surface reconstruction of the burden surface is achieved based on the improved PMVS and the Poisson surface reconstruction method. Experimental results demonstrate that the proposed method can extract accurate feature points and reconstruct the accurate 3D topography of the burden surface.

LOFF: LiDAR and Optical Flow Fusion Odometry

LOFF: LiDAR and Optical Flow Fusion Odometry

A Single-Pixel Event Photoactive Device for Real-Time, In-Sensor Spatiotemporal Optical Information Processing.

The increasing demand for energy-efficient, sophisticated optical sensing technologies in various applications, from machine vision to optical communication, highlights the necessity for innovations in spatiotemporal information sensing and processing at a nearly single-pixel scale. Traditional methods, including multi-pixel photodetector arrays and event-based camera systems, often fail to provide rapid, real-time detection and processing of dynamic events within the sensor. This shortcoming is particularly notable in handling high-dimensional spatiotemporal data, where the dependency on sequential data input and external processing tools leads to latency, reduced throughput, and heightened energy consumption, thereby impeding real-time parallel data processing capabilities. Here, a carrier-selective, single-pixel, position-sensitive planar photoactive device that integrates spatiotemporal event sensing with inherent short-term memory capabilities is introduced. The proof-of-concept single-pixel event photoactive device enables in-sensor spatiotemporal parallel optical information processing, efficiently managing multibit (>4 bit) data simultaneously and facilitating ultrafast (≈0.4µs) recognition of input patterns with low energy consumption (25 femtojoules). Additionally, by adjusting the operating speed from continuous to pulsed light illumination, the sensor array can detect trajectories and absolute position of events, offering in-sensor optical flow detection. This single-pixel event photodetector marks significant advancement toward developing compact, energy-efficient, ultrafast sensors suitable for a wide range of in sensor-based photonic applications.

A multi-sensor fusion SLAM algorithm for indoor aerial robots

Nowdays, existing Light Detection and Ranging (LiDAR) based unknown-space-exploration Simultaneous Localization and Mapping (SLAM) methods have the problems of unstable mapping in feature-degraded environments and poor estimation accuracy in the vertical direction. To solve these problems, this study presents a multi-sensor fusion SLAM algorithm for indoor aerial robots as well as Unmanned Aerial Vehicles (UAVs). The sensors used in the multi-sensor fusion algorithm include optical flow sensor, barometer, Inertial Measurement Unit (IMU), and LiDAR. The proposed algorithm is named LiDAR-IMU-optical flow odometry fusion algorithm (Lioo). We derived and established a tightly coupled multi-sensor fusion framework based on factor graph optimisation. The IMU-optical flow pre-integration factor is proposed, and the barometer factor to the algorithm is added innovatively. Engineering applications are also realised in this paper. Flight experiments demonstrated that the proposed algorithm achieves more precise mapping and localisation of UAVs in indoor environments.

MREIFlow: Unsupervised dense and time-continuous optical flow estimation from image and event data

Objectives:We focus on exploring an unsupervised learning-based model which can take advantage of a single image and events to estimate dense and time-continuous optical flow. Methods:We propose a multi-scale optical flow recurrent estimation network, called MREIFlow, which mainly consists of a triplet feature encoder, a feature fusion subnetwork, and a flow iterative decoder. The triplet feature encoder is capable of extracting multi-scale features from a single image and events. The feature fusion subnetwork is designed to integrate spatial–temporal information in features and generate pseudo-features. The flow iterative decoder can perform feature correlation and estimate full-resolution optical flow iteratively in a coarse-to-fine way. We train the network in an unsupervised learning way to avoid using labeled data. In order to provide plentiful and reliable supervisory signals, we design a hierarchical unsupervised learning scheme. In the scheme, the multi-level optimization objectives are designed to supervise network training in different aspects. We also introduce a teacher model to provide auxiliary supervision. Besides, we apply a loss selection strategy to mine promising supervisory signals. Results:Experiments on the MVSEC dataset show that the proposed method achieves remarkable performance under indoor and outdoor circumstances. Compared with previous methods, the proposed method can provide dense and time-continuous optical flow estimation. Conclusion:The proposed network can take advantage of a single image and events to produce dense and time-continuous optical flow estimation, and can be trained through unsupervised learning techniques. However, it requires high computation and memory to guarantee the precision of optical flow. To address this problem, we will explore an efficient and lightweight network architecture in the future.

Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flow

This paper presents a production-oriented 4D facial reconstruction pipeline designed to produce high-fidelity facial mesh sequences with a consistently structured topology, while preserving the wireframe structure specified by artists. We have designed and developed a compact, efficient, and fast optical capture system based on synchronized camera arrays for high-precision dynamic 3D facial imaging. Unlike prevailing methods that primarily concentrate on single-frame reconstruction, often reliant on labor-intensive manual annotation, our framework exploits the constraint of appearance consistency to autonomously establish feature correspondence and uphold temporal coherence within the mesh. Consequently, our approach eliminates mesh drifting and jitter, enabling full parallelization for dynamic facial expression capture. The proposed pipeline decouples the non-linear deformation of facial expressions from the rigid movements of the skull through a stable external device. Leveraging progressive retopology, our methodology employs artist-guided templates as priors, ensuring the preservation of wireframe structures across the result sequence. Progressive retopology is achieved by constraining different fine-grained features of 3D landmarks, scan surface shapes, and appearance textures. The results of our study showcase facial mesh sequences with production-quality topology, adept at faithfully reproducing character expressions from photographs while achieving artist-friendly stable facial movements.

Three-view cotton flower counting through multi-object tracking and RGB-D imagery

Monitoring the number of cotton flowers can provide important information for breeders to assess the flowering time and the productivity of genotypes because flowering marks the transition from vegetative growth to reproductive growth and impacts the final yield. Traditional manual counting methods are time-consuming and impractical for large-scale fields. To count cotton flowers efficiently and accurately, a multi-view multi-object tracking approach was proposed by using both RGB and depth images collected by three RGB-D cameras fixed on a ground robotic platform. The tracking-by-detection algorithm was employed to track flowers from three views simultaneously and remove duplicated counting from single views. Specifically, an object detection model (YOLOv8) was trained to detect flowers in RGB images and a deep learning-based optical flow model Recurrent All-pairs Field Transforms (RAFT) was used to estimate motion between two adjacent frames. The intersection over union and distance costs were employed to associate flowers in the tracking algorithm. Additionally, tracked flowers were segmented in RGB images and the depth of each flower was obtained from the corresponding depth image. Those flowers tracked with known depth from two side views were then projected onto the middle image coordinate using camera calibration parameters. Finally, a constrained hierarchy clustering algorithm clustered all flowers in the middle image coordinate to remove duplicated counting from three views. The results showed that the mean average precision of trained YOLOv8x was 96.4%. The counting results of the developed method were highly correlated with those counted manually with a coefficient of determination of 0.92. Besides, the mean absolute percentage error of all 25 testing videos was 6.22%. The predicted cumulative flower number of Pima cotton flowers is higher than that of Acala Maxxa, which is consistent with what breeders have observed. Furthermore, the developed method can also obtain the flower number distributions of different genotypes without laborious manual counting in the field. Overall, the three-view approach provides an efficient and effective approach to count cotton flowers from multiple views. By collecting the video data continuously, this method is beneficial for breeders to dissect genetic mechanisms of flowering time with unprecedented spatial and temporal resolution, also providing a means to discern genetic differences in fecundity, the number of flowers that result in harvestable bolls. The code and datasets used in this paper can be accessed on GitHub: https://github.com/UGA-BSAIL/Multi-view_flower_counting.

Bio-Inspired Strategies Are Adaptable to Sensors Manufactured on the Moon.

Bio-inspired strategies for robotic sensing are essential for in situ manufactured sensors on the Moon. Sensors are one crucial component of robots that should be manufactured from lunar resources to industrialize the Moon at low cost. We are concerned with two classes of sensor: (a) position sensors and derivatives thereof are the most elementary of measurements; and (b) light sensing arrays provide for distance measurement within the visible waveband. Terrestrial approaches to sensor design cannot be accommodated within the severe limitations imposed by the material resources and expected manufacturing competences on the Moon. Displacement and strain sensors may be constructed as potentiometers with aluminium extracted from anorthite. Anorthite is also a source of silica from which quartz may be manufactured. Thus, piezoelectric sensors may be constructed. Silicone plastic (siloxane) is an elastomer that may be derived from lunar volatiles. This offers the prospect for tactile sensing arrays. All components of photomultiplier tubes may be constructed from lunar resources. However, the spatial resolution of photomultiplier tubes is limited so only modest array sizes can be constructed. This requires us to exploit biomimetic strategies: (i) optical flow provides the visual navigation competences of insects implemented through modest circuitry, and (ii) foveated vision trades the visual resolution deficiencies with higher resolution of pan-tilt motors enabled by micro-stepping. Thus, basic sensors may be manufactured from lunar resources. They are elementary components of robotic machines that are crucial for constructing a sustainable lunar infrastructure. Constraints imposed by the Moon may be compensated for using biomimetic strategies which are adaptable to non-Earth environments.