Camera Shake Research Articles

Estimation of passion fruit yield based on YOLOv8n + OC-SORT + CRCM algorithm

The precise estimation of passion fruit yield is crucial for efficient orchard management, but it poses challenges such as occlusion, light variations, and camera shake, which can lead to problems such as missed detection, error detection, and double counting of small fruits. In this study, we propose a robust computer vision algorithm named YOLOv8n + OC-SORT + CRCM (Central Region Counting Method) to accomplish three tasks: detection, tracking, and yield estimation of passion fruits. Firstly, we compare the passion fruit detection results using various YOLO series detection algorithms and choose YOLOv8n for detector. Then, OC-SORT algorithm is chosen as the tracker due to its effectiveness in addressing issues like occlusion, vertical shaking, and uneven speeds. Finally, we design CRCM counting algorithm for fruit counting for addressing challenges in estimating passion fruit yield. To validate the effectiveness of these methods, a real-world passion fruit video dataset including 24 videos for each with a 1-minute length was established. In the detection results on the test set, YOLOv8n detector achieved the best results with a mAP@0.5 (mean Average Precision) of 86.3 % and a model size of only 6.2 MB among YOLOv5n, YOLOv7 and YOLOv8n three detectors. Furthermore, the HOTA (higher order tracking accuracy) of OC-SORT tracker was 67.10 %, surpassing three mainstream trackers named BoT-SORT, Byte Track, and Strong SORT by 2.98 %, 4.71 %, and 8.82 %, respectively. In the fruit yield estimation, CRCM demonstrated an average counting accuracy of 87.0 %, surpassing ID number and Single Line Method (SLM) methods by 49.8 % and 10.5 %, respectively. In conclusion, the YOLOv8n + OC-SORT + CRCM algorithm effectively addresses issues of misidentification, missed detections of small fruits, and repeated counts, achieving stable, real-time, and accurate estimation of passion fruit yield.

Motion adaptive vision-based vibration measurement and modal identification for the roof masts of a tall building

On May 18, 2021, occupants in Saige Plaza Building felt significant building motions together with its roof masts caught in obvious vibrations (May 18 vibration event), which were recorded by a surveillance camera installed on the roof of the building. This study aims to investigate the vibration characteristics of masts by processing the video data using computer vision technique. A motion adaptive vision-based vibration measurement method (M-DAVIM) is firstly proposed, focusing on dealing with the adverse motion effects of the camera itself on measuring dynamic displacements. The M-DAVIM incorporates time-domain and frequency domain correction procedures to reduce noise caused by camera self-vibration, and utilizes a CNN-based object tracking method to search objects that blurred by camera shaking. Indoor periodic vibration tests and field tests of photovoltaic panels demonstrated that M-DAVIM outperforms previous vision-based methods in accurately measuring displacements under unfavorable conditions, such as target rotation, motion blurring, and background interference. The proposed M-DAVIM was then applied to measure the dynamic displacements of the roof masts of Saige Building and identify their modal parameters (frequencies and damping ratios) based on the limited video data. A vibration component of 7.60 Hz was identified as the camera self-vibration and was effectively corrected by the M-DAVIM method. Based on finite element analysis and given the wind condition, the twin-mast might mainly experience the vortex-induced vibration at 2.12 Hz with two masts vibrating synchronously in-plane along opposite directions during May 18 to 22, 2021. This study demonstrates the robustness and effectiveness of the M-DAVIM and shows its potential application for long-term field monitoring of large-scale structures under severe outdoor environments.

NIMG-77. MRI PHENOMENON OF PUNCTATE ENHANCING LESIONS IN PATIENTS TREATED WITH IDH INHIBITORS

Abstract BACKGROUND Magnetic Resonance Imaging (MRI) response assessment is essential in isocitrate dehydrogenase (IDH)-mutant gliomas and provides evidence of treatment efficacy. Ivosidenib, a mutant IDH1 inhibitor, and vorasidenib, a pan-mutant IDH inhibitor, may induce imaging stabilization or improvement. We describe an imaging phenomenon of punctate enhancing lesions in patients receiving ivosidenib or vorasidenib. METHODS From October 2020 to December 2023, we identified patients treated with IDH inhibitors who developed punctate enhancing lesions on imaging during their treatment course. We report their clinical and imaging characteristics. RESULTS In our cohort (n=5), patients age range was 29-47 years. Diagnoses included astrocytoma grades 2 (n=2) or 3 (n=1), and oligodendroglioma grades 2 (n=1) or 3 (n=1). Beside all having a prior resection, two had prior radiation therapy and temozolomide, one had prior temozolomide only and two were treatment-naïve. Before initiation of IDH inhibitor, all patients had some evidence of enhancement, albeit small (<1 cm) for most. Four patients received ivosidenib (500mg daily) and one received vorasidenib (40mg daily). The median time to develop punctate lesions from treatment initiation was 16 months (range 1-28 months). All but one lesion was < 1 cm and adjacent to the original tumor locations. Four had FLAIR correlation, and two had high DWI signal without associated restricted diffusion. Punctate lesions resolved on subsequent imaging, within 1-2 months, for all grade 2 tumors. Punctate lesions persisted in the grade 3 tumors, without further progression. Four patients remain on therapy. One discontinued treatment due to this enhancement, which resolved on subsequent imaging. CONCLUSIONS Punctate enhancing lesions may develop in patients on IDH inhibitors. We advocate for close interval imaging monitoring in these cases since they may not universally require treatment discontinuation. Future advanced imaging and tissue evaluations of punctate enhancement, to differentiate between inflammatory response and progression, would be of interest.

MONOCHROME AUGMENTED LOW-LIGHT IMAGE ENHANCEMENT

Low light short exposure photography is challenging, but an important factor in capturing images in temporarily dynamic scenes avoiding unwanted effects such as ghosting, motion blur, camera shakes, image artifacts, etc. Monochrome augmented low-light image enhancement aims to get improved low-light short-exposure images by using an additional monochrome sensor and its data. Monochrome images typically possess a higher SNR (Signal-to-Noise Ratio) and better luma information, since it avoids the attenuation by the Bayer Filter. The objective here is to develop a deep learning based approach to enhance low-light short exposure images from the main sensor by using an additional low resolution monochrome sensor.

An effective object detection algorithm for UAV-based urban regulation

Target detection from the perspective of UAV has great potential in the field of urban regulation, limited by the dense small targets, severe environmental obstructions, camera shake, and changes in lighting conditions in the aerial view of drones, the existing object detection algorithms cannot effectively undertake this task. This paper introduces two lightweight feature extraction modules based on YOLOv5, which are C3-Faster with PConv and COT3 with transformer structure. Meanwhile, an extra small detection head is added to the output layer. These approaches enhance accuracy while maintaining the advantages of being lightweight and easy to deploy. The ablation experiments and comparative experiments are designed to verify the effectiveness of these modules. The algorithm presented in this paper can be deployed into embedded systems of small UAVs to assist UAVs in completing various regulatory tasks in complex urban scenarios.

Tightly coupled visual-inertial fusion with image enhancement for robust positioning

Abstract Traditional vision-based inertial odometry (VIO) suffers from significant visual degradation, which substantially impacts state estimation in challenging lighting environments. Thermal imaging cameras capture images based on the thermal radiation of objects, rendering them impervious to lighting variations. However, integrating thermal infrared data into conventional visual odometry poses challenges due to its low texture, poor contrast, and high noise levels. In this paper, we propose a tightly coupled approach that seamlessly integrates information from visible light cameras, thermal imaging cameras, and inertial measurement units (IMUs). First, we employ adaptive bilateral filtering and Sobel gradient enhancement to smooth infrared images, thereby reducing noise and enhancing edge contrast. Second, we leverage the Sage-Husa adaptive filter in conjunction with iterative Kalman filtering (IEKF) to effectively mitigate the impact of non-Gaussian noise on the system. Finally, we conduct comprehensive evaluations of the proposed system using both open datasets and real-world experiments across four distinct scenarios: normal lighting, low-light conditions, low-light conditions with camera shake, and challenging lighting environments. Comparative analysis reveals that our method outperforms IEKF, yielding a reduction in localization error measured by root mean square error (RMSE) by 58.69%, 57.24%, 60.23%, and 30.87% in these respective scenarios.

Single Image Integrated Deblurring Algorithm in Non-Uniform Environment

Motion blurriness in an image caused by camera shake during exposure is unavoidable. It could lead to information loss and degradation in the image quality. Therefore, many researchers are dedicated to developing image deblurring techniques to recover clear images from blurred images. During the deblurring process, structural edges in images play a vital part in estimating the blur kernels. For images with rich textures, fine-scale edges become more apparent. This will cause vagueness in the image’s structural edges and affect the accuracy of the kernel estimation process. In this study, we propose a single-image motion deblurring by kernel estimation method combined with L0-Regularized Intensity and Gradient Prior, and enhanced Scale Aware Smoothing methods. Two types of non-uniform datasets are used, which are real and synthetic. While synthetic datasets are utilized to assess the consistency in performance across real and synthetic images, real datasets are used to portray the level of detail and variation of actual blurred images. The dataset is divided into five categories (people, nature, manmade, text, and night light). Two image quality metrics were selected: full-reference assessment, including learned perceptual image patch similarity (LPIPS), peak signal-to-noise ratio (PSNR), and structural similarity indexing method (SSIM) for synthetic datasets, and no-reference assessments, including blind/reference-less image spatial quality (BRISQUE), natural image quality evaluator (NIQE) and perception-based image quality evaluator (PIQE) for real datasets. According to the findings, the fusion method performs best in the text category, followed by manmade and in nature, then night light and poor in people. The proposed method not only removes fine-scale edges and preserves the boundary sharpness, but also improved the estimated blur kernel.

Multi-Object Tracking Model Based on Detection Tracking Paradigm in Panoramic Scenes

Multi-Object Tracking (MOT) technology is dedicated to continuously tracking multiple targets of interest in a sequence of images and accurately identifying their specific positions at different times. This technology is crucial in key application areas such as autonomous driving and security surveillance. However, the application process often requires the coordination of cameras from multiple angles for tracking. Directly studying Multi-Object Tracking algorithms in panoramic scenes is an effective way to address this issue. The uniqueness of panoramic scenes causes target position changes at the boundaries and tracking difficulties due to continuous changes in target scales. To ensure the accuracy of target tracking, this study explores a detection-based tracking method using the newly improved YOLOx detector and the adjusted DeepSORT algorithm. Firstly, YOLOx_s was chosen as the detector because its simple network structure ensures a fast computational speed. During the feature extraction stage, we used the Polarized Self-Attention (PSA) mechanism to capture more feature information, thereby improving the tracking performance on small-scale targets. Secondly, the tracker was improved by adding a camera motion compensation module before predicting the target’s position to mitigate the impact of camera shake on tracking. Finally, to address the difficulty of continuously tracking targets in specific areas of panoramic scenes, this study proposes specific tracking strategies. These strategies effectively resolve the problem of tracking failure caused by target position changes at the boundaries. Experimental results show that improved algorithms have a superior performance on multiple evaluation metrics compared to other algorithms in the field. Compared to the original algorithm, the improved algorithm exhibits a 6% increase in the quantitative metric MOTA, a 7% increase in IDF1, and a 40% decrease in IDSWs, demonstrating its leading performance.

Iterative feedback-based models for image and video polyp segmentation

Accurate segmentation of polyps in colonoscopy images has gained significant attention in recent years, given its crucial role in automated colorectal cancer diagnosis. Many existing deep learning-based methods follow a one-stage processing pipeline, often involving feature fusion across different levels or utilizing boundary-related attention mechanisms. Drawing on the success of applying Iterative Feedback Units (IFU) in image polyp segmentation, this paper proposes FlowICBNet by extending the IFU to the domain of video polyp segmentation. By harnessing the unique capabilities of IFU to propagate and refine past segmentation results, our method proves effective in mitigating challenges linked to the inherent limitations of endoscopic imaging, notably the presence of frequent camera shake and frame defocusing. Furthermore, in FlowICBNet, we introduce two pivotal modules: Reference Frame Selection (RFS) and Flow Guided Warping (FGW). These modules play a crucial role in filtering and selecting the most suitable historical reference frames for the task at hand. The experimental results on a large video polyp segmentation dataset demonstrate that our method can significantly outperform state-of-the-art methods by notable margins achieving an average metrics improvement of 7.5% on SUN-SEG-Easy and 7.4% on SUN-SEG-Hard. Our code is available at https://github.com/eraserNut/ICBNet.

MR Imaging Stabilization of Leukodystrophy-like MOG-associated Disease in a Patient Treated with Rituximab (P9-14.019)

MR Imaging Stabilization of Leukodystrophy-like MOG-associated Disease in a Patient Treated with Rituximab (P9-14.019)

Decision-making ability limitations and brain neural activity changes in healthcare workers after mild COVID-19

Studies have demonstrated that the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) extensively affects brain function. Although cognitive dysfunction is considered a common manifestation in COVID-19 patients during the recovery period, the potential changes in decision-making ability, are not yet clear. Decision-making functions are essential to the work of healthcare workers. However, there is a lack of a multidimensional assessment of its functioning in COVID-19 cases. Here, we used tests combined with the resting-state functional magnetic resonance imaging (rs-fMRI) stabilization feature amplitude of low-frequency fluctuations (ALFF) to explore decision-making behavior and brain neural activity changes in healthcare workers after mild COVID-19. Participants were divided into the SARS-CoV-2 infected group (SI, n = 41) and healthy controls (HC, n = 42). All participants underwent a series of neuropsychological tests. They performed the Iowa Gambling Task (IGT) and the Game of Dice Task (GDT), followed by fMRI (n = 20) to assess their decision-making ability under ambiguous and risky conditions and changes in brain neural activity. The SI group performed worse in verbal memory than the HC group. Furthermore, the SI group performed worse in the IGT, whereas no significant difference was observed in the GDT. In addition, rs-fMRI showed enhanced spontaneous neural activity in the postcentral gyrus and inferior parietal lobe in the SI group compared to the HC group.

TSFF-Net: A novel lightweight network for video real-time detection of SF6 gas leaks

Detecting Sulphur hexafluoride (SF6) gas leaks using infrared video images is the latest technology. However, the current algorithm for autonomous real-time detection is imprecise and incapable of performing the real-time monitoring task effectively. In response to the aforementioned issues, we present a temporal–spatial feature fusion network (TSFF-Net) specifically designed for video real-time detection of SF6 gas leaks. Compared with the existing SF6 gas leak detection model based on deep learning, the biggest feature of TSFF-Net is the introduction of foreground pixel images with SF6 gas movement characteristics, which greatly enhances the model’s ability to detect SF6 gas leaks. Simultaneously, by designing a new attention feature fusion module, introducing the Neck module of motion features, and the Backbone module to enhance small target feature extraction, we achieve high-precision SF6 gas leak detection in various complex scenarios, such as sparse leaks, complex backgrounds, camera shake, etc. The mean average precision (mAP) of TSFF-Net in the validation set was 56.3%, which is 13.2% higher than that of Yolov8s. In addition, the model proposed in this paper has parameters that are only 2.6 M, making it suitable for deployment on mobile or embedded devices. The code and test results are at https://github.com/1jianyao/TSFF-Net/tree/main.

CNB Net: A Two-Stage Approach for Effective Image Deblurring

Image blur, often caused by camera shake and object movement, poses a significant challenge in computer vision. Image deblurring strives to restore clarity to these images. Traditional single-stage methods, while effective in detail enhancement, often neglect global context in favor of local information. Yet, both aspects are crucial, especially in real-life scenarios where images are typically large and subject to various blurs. Addressing this, we introduce CNB Net, an innovative deblurring network adept at integrating global and local insights for enhanced image restoration. The network operates in two stages, utilizing our specially designed Convolution and Normalization-Based Block (CNB Block) and Convolution and Normalization-Based Plus Block (CNBP Block) for multi-scale information extraction. A progressive learning approach is adopted with a Feature Active Selection (FAS) module at the end of each stage that captures spatial detail information under the guidance of real images. The Two-Stage Feature Fusion (TSFF) module reduces information loss caused by downsampling operations while enriching features across stages for increased robustness. We conduct experiments on the GoPro dataset and the HIDE dataset. On the GoPro dataset, our Peak Signal-to-Noise Ratio (PSNR) result is 32.21 and the Structural Similarity (SSIM) result is 0.950; and on the HIDE dataset, our PSNR result is 30.38 and the SSIM result is 0.932. Our results exceed other similar algorithms. By comparing the generated feature maps, we find that our model takes into account both global and local information well.

IMU-Assisted Accurate Blur Kernel Re-Estimation in Non-Uniform Camera Shake Deblurring.

Image deblurring for camera shake is a highly regarded problem in the field of computer vision. A promising solution is the patch-wise non-uniform image deblurring algorithms, where a linear transformation model is typically established between different blur kernels to re-estimate poorly estimated blur kernels. However, the linear model struggles to effectively describe the nonlinear transformation relationships between blur kernels. A key observation is that the inertial measurement unit (IMU) provides motion data of the camera, which is helpful in describing the landmarks of the blur kernel. This paper presents a new IMU-assisted method for the re-estimation of poorly estimated blur kernels. This method establishes a nonlinear transformation relationship model between blur kernels of different patches using IMU motion data. Subsequently, an optimization problem is applied to re-estimate poorly estimated blur kernels by incorporating this relationship model with neighboring well-estimated kernels. Experimental results demonstrate that this blur kernel re-estimation method outperforms existing methods.

Machine Learning‐Based Distraction‐Free Method for Measuring the Optical Displacement of Long‐Span Bridge Structures

A photogrammetric displacement measurement method based on machine learning was proposed to improve the robustness to environmental disturbances. (1) To reduce the target positioning error caused by environmental vibration especially atmospheric turbulence, a machine learning‐based weighted location algorithm combined with an adaptive window selection strategy was developed. In an outdoor displacement table experiment, the proposed method’s root mean squared error (RMSE) is 0.04 mm when the distance is 50 m, showing better accuracy and stability. (2) To complement/correct the missing or anomalous data caused by adverse external conditions, such as severe occlusion or camera shaking, a fast data self‐diagnosis using a correlation vector machine was performed to make full use of the full‐field measurement results obtained from the images. The applicability of the proposed method in extreme cases was demonstrated in designed experiments and an actual displacement measurement task of a long‐span bridge subjected to vortex‐induced vibration.

Vision-based approach for the static and dynamic monitoring of bridges

Structural Health Monitoring (SHM) is one of the main approaches to deal with damage identification in existing bridges. Static information together with structure modal properties allow to prevent collapses, detect damage also in the early stage, and plan maintenance works based on the bridge condition. Measurement systems are traditionally composed of a network of sensors directly installed on the structure. Despite the large diffusion of these systems, the expensive and time-consuming installation of sensors and acquisition system makes their use not always feasible. A promising approach for the characterization of bridge dynamic behavior is represented by computer vision-based techniques, which require the sole installation of one or more cameras outside the structure, along with some targets on it when necessary. This approach is totally non-invasive, low-cost and enables the direct measurement of structural displacements, providing useful and direct information about the operational conditions and possible permanent deformations. With the aim of investigating the potential of vision-based techniques for the dynamic monitoring of structures, this paper presents preliminary results of dynamic tests performed on a steel footbridge. Structural vibrations caused by a jumping pedestrian were measured from a camera placed at the riverbed as well as by an accelerometer-based monitoring system installed for validation purposes. The post-processing of video recordings is here presented and discussed, with particular emphasis on the impact of target shape and camera shaking.

INformer: Inertial-Based Fusion Transformer for Camera Shake Deblurring.

Inertial measurement units (IMU) in the capturing device can record the motion information of the device, with gyroscopes measuring angular velocity and accelerometers measuring acceleration. However, conventional deblurring methods seldom incorporate IMU data, and existing approaches that utilize IMU information often face challenges in fully leveraging this valuable data, resulting in noise issues from the sensors. To address these issues, in this paper, we propose a multi-stage deblurring network named INformer, which combines inertial information with the Transformer architecture. Specifically, we design an IMU-image Attention Fusion (IAF) block to merge motion information derived from inertial measurements with blurry image features at the attention level. Furthermore, we introduce an Inertial-Guided Deformable Attention (IGDA) block for utilizing the motion information features as guidance to adaptively adjust the receptive field, which can further refine the corresponding blur kernel for pixels. Extensive experiments on comprehensive benchmarks demonstrate that our proposed method performs favorably against state-of-the-art deblurring approaches.

A defocus and similarity attention-based cascaded network for multi-focus and misaligned image fusion

Multi-focus image fusion uses multiple images focused on different depths to generate a clear image covering the whole scene. The existing multi-focus image fusion methods do not consider the defocus successive variation in close-range photography and the camera shaking in sequence image shooting, and most methods cannot process multiple images simultaneously. We proposed an end-to-end deep learning network to generate the all-in-focus image from multi-focus and misaligned images to solve these problems. Specifically, taking multiple multi-focus and misaligned source images as input, our Defocus and Similarity Attention Fusion Network (DSAF-Net) first generates the corresponding defocus map through the Defocus-Net, and warps the source images and defocus maps to a unified camera view according to the optical flow estimated by the OpticalFlow-Net. Finally, our model applies a coarse-to-fine correspondence matching scheme to obtain the similarity weight maps, combined with the warped source images and defocus maps to fuse into a clear image. For training and testing our DSAF-Net, a multi-focus and misaligned cultural heritage photography dataset (WHU-MFM) is constructed using Blender Cycles renderer. Experiments results demonstrate that our method outperforms the state-of-the-art methods both qualitatively and quantitatively. DSAF-Net is available at https://github.com/PeimingCHEN/DSAF-Net. And WHU-MFM is available at https://github.com/PeimingCHEN/WHU-MFM-Dataset.

Image deblurring: comparison and analysis

Technological advancements and the advent of digital devices and media make images an important part of today’s social life. Image blurring is a common challenge that results from multiple factors such as object movement, camera shake, and raindrops, among others. Image deblurring has progressively become an important field of image restoration as directed by research findings. After research for more than five decades, significant research efforts have yielded useful technologies of image deblurring. This article provides an overview of the current knowledge on image deblurring technology by focusing on the classical methods and modern trends in the field. The article reviews the conventional methods and achievements made in past studies using evidence from 34 scholarly articles. The article also examines the application of algorithms in specific deblurring methodologies adopted in recent works. It covers the recent trend of learning-based models used to restore images and their effectiveness. They include Convolutional Neural Networks, Recurrent Neural Networks and Graph Convolutional Networks. Novel deep-learning deblurring techniques are also explored. Based on the findings, issues of concerns, opportunities and direction for future research are provided to advance image deblurring technologies.

Towards object tracking for quadruped robots

With the development of quadruped robot technology, object tracking for quadruped robots has become an important research topic where violent camera shaking caused by the robot’s movement makes this task very challenging. In this letter, a quadruped robot object tracking dataset (QROD-111), including 111 video sequences, is first established, which was collected through our quadruped robot platform. A tracking algorithm based on Siamese network is then proposed where an alignment module is introduced to alleviate tracking difficulties caused by the quadruped robot movement. Moreover, a scale adaptation subnetwork is designed to alleviate the impact of the object scale variation during the whole tracking process. Experimental results demonstrate that our algorithm can achieve advanced performance for quadruped robot object tracking.