Nighttime Video Research Articles

Road Traffic Classification from Nighttime Videos Using the Multihead Self-Attention Vision Transformer Model and the SVM

Road Traffic Classification from Nighttime Videos Using the Multihead Self-Attention Vision Transformer Model and the SVM

NightRain: Nighttime Video Deraining via Adaptive-Rain-Removal and Adaptive-Correction

Existing deep-learning-based methods for nighttime video deraining rely on synthetic data due to the absence of real-world paired data. However, the intricacies of the real world, particularly with the presence of light effects and low-light regions affected by noise, create significant domain gaps, hampering synthetic-trained models in removing rain streaks properly and leading to over-saturation and color shifts. Motivated by this, we introduce NightRain, a novel nighttime video deraining method with adaptive-rain-removal and adaptive-correction. Our adaptive-rain-removal uses unlabeled rain videos to enable our model to derain real-world rain videos, particularly in regions affected by complex light effects. The idea is to allow our model to obtain rain-free regions based on the confidence scores. Once rain-free regions and the corresponding regions from our input are obtained, we can have region-based paired real data. These paired data are used to train our model using a teacher-student framework, allowing the model to iteratively learn from less challenging regions to more challenging regions. Our adaptive-correction aims to rectify errors in our model's predictions, such as over-saturation and color shifts. The idea is to learn from clear night input training videos based on the differences or distance between those input videos and their corresponding predictions. Our model learns from these differences, compelling our model to correct the errors. From extensive experiments, our method demonstrates state-of-the-art performance. It achieves a PSNR of 26.73dB, surpassing existing nighttime video deraining methods by a substantial margin of 13.7%.

A narrative review on obstructive sleep apnoea syndrome in paediatric population.

Obstructive sleep apnoea syndrome is a respiratory sleep disorder that affects 1-5% of children. It occurs equally in males and females, with higher incidence in school age and adolescence. OSAS may be caused by several factors, but in children, adenotonsillar hypertrophy, obesity, and maxillo-mandibular deficits are the most common. In general, there is a reduction in the diameter of the airway with reduced airflow. This condition worsens during sleep due to the muscular hypotonia, resulting in apnoeas or hypoventilation. While snoring is the primary symptom, OSAS-related manifestations have a wide spectrum. Some of these symptoms relate to the nocturnal phase, including disturbed sleep, frequent changes of position, apnoeas and oral respiration. Other symptoms concern the daytime hours, such as drowsiness, irritability, inattention, difficulties with learning and memorisation, and poor school performance, especially in patient suffering from overlapping syndromes (e.g., Down syndrome). In some cases, the child's general growth may also be affected. Early diagnosis of this condition is crucial in limiting associated symptoms that can significantly impact a paediatric patient's quality of life, with the potential for the condition to persist into adulthood. Diagnosis involves evaluating several aspects, beginning with a comprehensive anamnesis that includes specific questionnaires, followed by an objective examination. This is followed by instrumental diagnosis, for which polysomnography is considered the gold standard, assessing several parameters, including the apnoea-hypopnoea index (AHI) and oxygen saturation. However, it is not the sole tool for assessing the characteristics of this condition. Other possibilities, such as night-time video recording, nocturnal oximetry, can be chosen when polysomnography is not available and even tested at home, even though with a lower diagnostic accuracy. The treatment of OSAS varies depending on the cause. In children, the most frequent therapies are adenotonsillectomy or orthodontic therapies, specifically maxillary expansion.

Nighttime Motorcycle Detection for Sparse Traffic Images Using Machine Learning

Traffic accidents often occur at night. It is understandable, since at night, people have low visibility. Many efforts to develop tools to detect nearby vehicles to avoid crashes have been reported. However, most of them worked only on detecting cars. The research aims to detect motorcycles at night, to complement the previous studies, which mainly focused on cars. The research introduces four features which are extracted from the red pixel and edge map. The algorithm to extract the features has also been developed. They are applied to three commonly used classifiers: Artificial Neural Network (ANN), Decision Tree, and Support Vector Machine (SVM) classifiers to validate the effectiveness of the features. Since the public dataset related to the research is not available yet, the nighttime videos from YouTube have been collected. The datasets contain all the various levels of darkness. They are divided into an 80-20 ratio for training and testing sets to support the experiment and measure the validity of the proposed method. As the best result, the detection using ANN can detect motorcycle proposals with accuracy of 72.71%, precision of 65.10% and recall of 73.33%. Furthermore, during the experiment, the classification can perform consistently in 0.04 seconds per image. Therefore, the method is suitable for use in a real-time system.

An Unpaired Thermal Infrared Image Translation Method Using GMA-CycleGAN

Automatically translating chromaticity-free thermal infrared (TIR) images into realistic color visible (CV) images is of great significance for autonomous vehicles, emergency rescue, robot navigation, nighttime video surveillance, and many other fields. Most recent designs use end-to-end neural networks to translate TIR directly to CV; however, compared to these networks, TIR has low contrast and an unclear texture for CV translation. Thus, directly translating the TIR temperature value of only one channel to the RGB color value of three channels without adding additional constraints or semantic information does not handle the one-to-three mapping problem between different domains in a good way, causing the translated CV images not only to have blurred edges but also color confusion. As for the methodology of the work, considering that in the translation from TIR to CV the most important process is to map information from the temperature domain into the color domain, an improved CycleGAN (GMA-CycleGAN) is proposed in this work in order to translate TIR images to grayscale visible (GV) images. Although the two domains have different properties, the numerical mapping is one-to-one, which reduces the color confusion caused by one-to-three mapping when translating TIR to CV. Then, a GV-CV translation network is applied to obtain CV images. Since the process of decomposing GV images into CV images is carried out in the same domain, edge blurring can be avoided. To enhance the boundary gradient between the object (pedestrian and vehicle) and the background, a mask attention module based on the TIR temperature mask and the CV semantic mask is designed without increasing the network parameters, and it is added to the feature encoding and decoding convolution layers of the CycleGAN generator. Moreover, a perceptual loss term is applied to the original CycleGAN loss function to bring the translated images closer to the real images regarding the space feature. In order to verify the effectiveness of the proposed method, the FLIR dataset is used for experiments, and the obtained results show that, compared to the state-of-the-art model, the subjective quality of the translated CV images obtained by the proposed method is better, as the objective evaluation metric FID (Fréchet inception distance) is reduced by 2.42 and the PSNR (peak signal-to-noise ratio) is improved by 1.43.

Study of Subjective and Objective Quality Assessment of Night-Time Videos

With the widespread usage of video capture devices and social media videos, videos are dominating the multimedia landscape. There is an emerging need for video quality assessment (VQA) that forms the backbone of advanced video systems. Night-time videos play an important role in user capturing, hence being able to accurately assess their quality is critical. However, the characteristics of night-time videos differ from those of general in-capture videos; and VQA algorithms that have been developed for general-purpose videos cannot accurately assess the quality of night-time videos. Research is needed to gain a better understanding of how humans perceive the quality of night-time videos, and use this new understanding to develop reliable VQA algorithms. To this end, we construct a large-scale night-time VQA database, namely Mobile In-capture Night-time Database for Video Quality (MIND-VQ), containing 1181 night-time videos, 435 subjects, and over 130000 opinion scores. We perform thorough analyses to reveal subjective quality assessment behaviors of night-time videos. Furthermore, we propose a new VQA model, namely Visibility-based Night-time Video Quality Assessment Network, VINIA. Spatial and temporal visibility-aware components are characterized to reflect properties of human perception of night-time VQA task. A series of experiments are conducted to compare our VINIA with other existing VQA algorithms using our new MIND-VQ database and other public VQA databases. Experimental results show that our subjective VQA database provides new insights and our new VINIA model achieves superior performance in accessing night-time video quality.

Investigation of Inlaid Pavement Marker Performance and Safety Effectiveness

The objectives of this study were to assess inlaid pavement marker (IPM) performance and estimate their safety effectiveness in St. Louis, Missouri. IPM performance was evaluated through a count of marker presence and a feedback survey from participants who viewed dry and wet night videos of IPM sections in the St Louis area. The safety effectiveness of IPMs was evaluated using a rigorous state-of-the-art empirical Bayes (EB) before–after crash analysis. The marker presence assessment, unfortunately, did not provide any conclusive trends in IPM performance. Newer IPM sections had higher percentages of missing markers than older sections, and there were too many unknown variables to determine the source of the unexpected differences. The unanimous conclusion of the nighttime video visibility survey was that drivers and passengers traveling on a wet night feel that IPMs are very important to the visibility of the roadway’s lane lines. For the EB analysis, installation and reference site data were used to examine the effects for specific crash types, including total, fatal and injury, wet pavement, nighttime, nighttime wet pavement, lane departure, wet pavement lane departure, and nighttime lane departure. Based on the aggregate results, IPMs, when installed with pavement resurfacing, significantly reduce all crash types examined.

Young drivers’ perception of hazards: Variation with experience and day versus night

Young novice drivers have a relatively high crash risk for several years following initial licensing, and while all drivers are at greater risk at night, the night-time increase is greater for inexperienced drivers. Poor hazard perception has been identified as an important contributor to inexperienced drivers’ risk, but research on day-night differences in hazard perception for drivers varying in experience is lacking. This exploratory study investigated the nature of hazards reported by young inexperienced drivers versus more experienced and slightly older drivers. Hazards were not pre-identified by researchers; participants were simply provided with a general definition of ‘hazard’. Analysis focused on how experience level affected qualitative differences in the kinds of hazards reported, with particular focus on day-night differences.The 53 participants ranged in driving experience from learners through to 5+years post licensing, and in age from 16 to 30 years. They viewed 14 day- and night-time video clips of a diverse range of driving situations, pausing the video whenever they identified a hazard and then explaining why they had paused it at that point. Their responses were recorded. Content analysis of responses showed that more experienced drivers reported visibility-related hazards significantly more often than inexperienced ones, and significantly more so at night. They also commented significantly more on hazards related to tight bends in the road and significantly less on hazards concerning compliance with rules. Comments tended to be fewer with higher vehicle speeds, particularly for the least experienced drivers.Results are discussed in terms of how experience-related differences in drivers’ cognitive schemata and mental models are likely to affect hazard perception and crash risk, particularly at night. Some implications for driver training and license testing are suggested.

Temporal Weighting Appearance-Aligned Network for Nighttime Video Retrieval

Video-based person re-identification (ReID) aims at re-identifying video sequences of a specified person from videos captured by disjoint cameras. Existing datasets and works on this task all focus on daytime scenarios and cannot adapt well to the nighttime scenarios, which is also of significant importance for practical applications. In this paper, we contribute a new dataset for nighttime video-based ReID, termed NIVIR, which contains 800 identities with over 228,000 images. NIVIR contains video shots under various lighting conditions, different weathers, and complex scenarios, which is consistent with the real nighttime outdoor surveillance. Furthermore, we propose a temporal weighting appearance-aligned network (TWAN) for nighttime video-based ReID, which is composed of a correlation-based appearance-aligned module (CAM) and a temporal weighting module (TWM). Specifically, CAM is proposed to reconstruct the adjacent feature maps to guarantee the appearance alignment between the central frame and its adjacent frames. TWM is designed to evaluate the frame quality of a tracklet and generate temporal weights to enhance the video representation. Extensive experiments conducted on our new NIVIR dataset demonstrate that the proposed TWAN outperforms the state-of-the-art methods. We believe that our NIVIR dataset and the comprehensive attempts for solving the nighttime ReID problem will push forward the development of the ReID research community.

Perceptual quality assessment of nighttime video

The quality of nighttime videos is important for consumer photography and monitoring of video clearness. However, little work has been done on the study of nighttime video quality assessment. In this paper, to the best of our knowledge, we explore the study on the nighttime video quality assessment for the first time. First, we build a real-world nighttime video quality assessment database (NVQA) containing 200 videos with abundant content and diverse distortion. Additionally, we carry out subjective tests to rate all nighttime videos in the NVQA database. Thereafter, we proposed a blind nighttime video quality assessment model based on feature fusion and conducted experiments to evaluate the performance and efficiency of our proposed model. The experiment results demonstrate that our model outperforms most traditional methods.

A vision-based method for automatic tracking of construction machines at nighttime based on deep learning illumination enhancement

Nighttime construction has been widely conducted in many construction scenarios, but it is also much riskier due to low lighting conditions and fatiguing environments. Therefore, this study proposes a vision-based method specifically for automatic tracking of construction machines at nighttime by integrating the deep learning illumination enhancement. Five main modules are involved in the proposed method, including illumination enhancement, machine detection, Kalman filter tracking, machine association, and linear assignment. Then, a testing experiment based on nine nighttime videos is conducted to evaluate the tracking performance using this approach. The results show that the method developed in this study achieved 95.1% in MOTA and 75.9% in MTOP. Compared with the baseline method SORT, the proposed method has improved the tracking robustness of 21.7% in nighttime construction scenarios. The proposed methodology can also be used to help accomplish automated surveillance tasks in nighttime construction to improve the productivity and safety performance.

Domain adaptation from daytime to nighttime: A situation-sensitive vehicle detection and traffic flow parameter estimation framework

Vehicle detection in traffic surveillance images is an important approach to obtain vehicle data and rich traffic flow parameters. Recently, deep learning based methods have been widely used in vehicle detection with high accuracy and efficiency. However, deep learning based methods require a large number of manually labeled ground truths (bounding box of each vehicle in each image) to train the Convolutional Neural Networks (CNN). In the modern urban surveillance cameras, there are already many manually labeled ground truths in daytime images for training CNN, while there are little or much less manually labeled ground truths in nighttime images. In this paper, we focus on the research to make maximum usage of labeled daytime images (Source Domain) to help the vehicle detection in unlabeled nighttime images (Target Domain). For this purpose, we propose a new situation-sensitive method based on Faster R-CNN with Domain Adaptation (DA) to improve the vehicle detection at nighttime. Furthermore, a situation-sensitive traffic flow parameter estimation method is developed based on the traffic flow theory. We collected a new dataset of 2,200 traffic images (1,200 for daytime and 1,000 for nighttime) of 57,059 vehicles to evaluate the proposed method for the vehicle detection. Another new dataset with three 1,800-frame daytime videos and one 1,800-frame nighttime video of about 260 K vehicles was collected to evaluate and show the estimated traffic flow parameters in different situations. The experimental results show the accuracy and effectiveness of the proposed method.

Assessment of laying-bird welfare following acaricidal treatment of a commercial flock naturally infested with the poultry red mite (Dermanyssus gallinae).

The poultry red mite (PRM), Dermanyssus gallinae, a potential vector of pathogens to animals and humans, causes impaired bird welfare. A study investigated changes in behavioural variables, physiological biomarkers, and health parameters following acaricidal treatment of PRM infestation of laying hens on a commercial farm. Mite traps determined the challenge to 12,700 hens before and after drinking water administration of the acaricide, fluralaner (Exzolt®, 0.5 mg/kg; Weeks 0 and 1). Weekly daytime direct observations and night-time video recordings monitored bird behaviours from Weeks -6 through +6. Blood samples were collected from randomly-selected birds (Weeks -6, -1, and +6). Following treatment, mite count reductions (>99%) were statistically significant (P < 0.0001), as were night-time reductions in the percent of hens showing activity, preening, head scratching (all P < 0.0001), and head shaking (P = 0.0007). Significant daytime reductions were observed in preening and head scratching (both P < 0.0001), head shaking (P = 0.0389), severe feather pecking (P = 0.0002), and aggressive behaviour (P = 0.0165). Post-treatment, comb wounds were significantly reduced (P = 0.0127), and comb colour was significantly improved (P < 0.0001). Heterophil/lymphocyte ratio was significantly reduced at Weeks 1 and 6 (P = 0.0009 and P < 0.0001, respectively). At Week 6, blood corticosterone (P = 0.0041) and total oxidant status (P < 0.0001) were significantly reduced, and haemoglobin and mean corpuscular haemoglobin significantly increased (P < 0.0001). Farm production records indicated that those post-treatment improvements were accompanied by significant reductions in weekly mortality rate (P = 0.0169), and significant recovery in mean weekly egg weights (P < 0.0001) and laying rate (P < 0.0001). The improvements in behavioural variables, physiological biomarkers, and health parameters that were observed following the elimination of PRM on a commercial farm indicate that infestations can be a cause of reduced hen welfare.

Texture collinearity foreground segmentation for night videos

One of the most difficult scenarios for unsupervised segmentation of moving objects is found in nighttime videos where the main challenges are the poor illumination conditions resulting in low-visibility of objects, very strong lights, surface-reflected light, a great variance of light intensity, sudden illumination changes, hard shadows, camouflaged objects, and noise. This paper proposes a novel method, coined COLBMOG (COLlinearity Boosted MOG), devised specifically for the foreground segmentation in nighttime videos, that shows the ability to overcome some of the limitations of state-of-the-art methods and still perform well in daytime scenarios. It is a texture-based classification method, using local texture modeling, complemented by a color-based classification method. The local texture at the pixel neighborhood is modeled as an N-dimensional vector. For a given pixel, the classification is based on the collinearity between this feature in the input frame and the reference background frame. For this purpose, a multimodal temporal model of the collinearity between texture vectors of background pixels is maintained. COLBMOG was objectively evaluated using the ChangeDetection.net (CDnet) 2014, Night Videos category, benchmark. COLBMOG ranks first among all the unsupervised methods. A detailed analysis of the results revealed the superior performance of the proposed method compared to the best performing state-of-the-art methods in this category, particularly evident in the presence of the most complex situations where all the algorithms tend to fail.

Enhancement of Night Time Video using Dark Channel Prior IP Accelerator

The enhancement of night-time video using Dark Channel Prior IP accelerator is proposed in this paper. Night time video processing is difficult due to low brightness, low contrast and high noise in the video. The above problems affect the accuracy and may results in failures of object detection in night time video. Dark Channel Prior (DCP) filter is used to improve the visibility, brightness and contrast of the video at night time. Processing speed is challenging task on real time application of DCP algorithm for night time video enhancement. Hence, DCP algorithm is implemented on FPGA (ALVEO Board) to increase the speed of video processing.

To See in the Dark: N2DGAN for Background Modeling in Nighttime Scene

Due to the deteriorated conditions of illumination lack and uneven lighting, the performance of traditional background modeling methods is greatly limited for the surveillance of nighttime video. To make background modeling under nighttime scene performs as well as in daytime condition, we put forward a promising generation-based background modeling framework for foreground surveillance. With a pre-specified daytime reference image as background frame, the GAN based generation model, called N2DGAN, is trained to transfer each frame of nighttime video to a virtual daytime image with the same scene to the reference image except for the foreground part. Specifically, to balance the preservation of background scene and the foreground object(s) in generating the virtual daytime image, we presented a two-pathway generation model, in which the global and local sub-networks were well combined with spatial and temporal consistency constraints. For the sequence of generated virtual daytime images, a multi-scale Bayes model was further proposed to characterize pertinently the temporal variation of background. We manually labeled ground truth on the collected nightime video datasets for performance evaluation. The impressive results illustrated in both the main paper and supplementary show the effectiveness of our proposed approach.

Improving Night-Time Pedestrian Retrieval With Distribution Alignment and Contextual Distance

Night-time pedestrian retrieval is a cross-modality retrieval task of retrieving person images between day-time visible images and night-time thermal images. It is a very challenging problem due to modality difference, camera variations, and person variations, but it plays an important role in night-time video surveillance. The existing cross-modality retrieval usually focuses on learning modality sharable feature representations to bridge the modality gap. In this article, we propose to utilize auxiliary information to improve the retrieval performance, which consistently improves the performance with different baseline loss functions. Our auxiliary information contains two major parts: cross-modality feature distribution and contextual information. The former aligns the cross-modality feature distributions between two modalities to improve the performance, and the latter optimizes the cross-modality distance measurement with the contextual information. We also demonstrate that abundant annotated visible pedestrian images, which are easily accessible, help to improve the cross-modality pedestrian retrieval as well. The proposed method is featured in two aspects: the auxiliary information does not need additional human intervention or annotation; it learns discriminative feature representations in an end-to-end deep learning manner. Extensive experiments on two cross-modality pedestrian retrieval datasets demonstrate the superiority of the proposed method, achieving much better performance than the state-of-the-arts.

Nighttime visual refinement techniques for surveillance video: a review

Video surveillance systems substitute manual efforts in various safety critic domains such as border area, assisted living, banking, service stations, and transportation. The multimedia-based surveillance system has a significant role in security and forensic systems because people tend to be easily convinced after observing voice, image, and video. Hence, these videos are strong evidence in the forensic investigation. However, most of the criminal activities such as ATM robbery and assassination are occur at nighttime because of the crime supporting dark environment. Many of the night surveillance systems in military, as well as commercial applications, are equipped with infrared and thermal based night vision systems. Its poor capability of texture and color interpretations are the major issues to ensure secure nighttime video monitoring. Specifically, visual refinements of nighttime surroundings and foreground objects provide a valuable assistance in the nighttime security system. In this scenario, it is highly recommended a review of the state-of-the-art nighttime visual refinement approaches. We conducted an extensive literature review and classified the nighttime visual refinement approaches into nighttime restoration and enhancement. This comparative literary analysis identified the research gap fields to explore future research directions in nighttime visual enhancement techniques. Finally, we discussed various open issues and future directions in the context enhancement based nighttime enhancement research.

Ship Detection and Tracking in Nighttime Video Images Based on the Method of LSDT

Ship detection and tracking has been recognized as a challenging task in the maritime administration. The method of LSDT (Light Spot Detection and Tracking) has been applied to achieve ship detection and tracking based on the nighttime surveillance video. Firstly, the light spots in the video images are detected through LOG and invalid spots are filtered by the gray threshold. Multiple targets are subsequently tracked by Kalman filtering and light spots are marked to determine properties in order to add and delete spots. A case study has been performed in the middle reach (Wuhan) of the Yangtze River. Nighttime surveillance video has been obtained and further utilized for method verification. The results show that the ship can be detected and tracked by LSDT effectively. The present study provides an alternative for ship detection at night, thus improving the maritime surveillance efficiency.

The effectiveness of night-time video EEG monitoring in nocturnal enuresis is low

The effectiveness of night-time video EEG monitoring in nocturnal enuresis is low