Video Monitoring Research Articles

HiFormer: Hierarchical transformer for grounded situation recognition

The prevalence of monitoring video is critical to public safety, but existing Object Detection and Action Recognition models are overwhelmed by camera operators, unable to identify relevant events. In light of this, Grounding Situation Recognition (GSR) provides a practical solution to recognize the events in a surveillance video. That is, GSR can identify the noun entities (e.g., humans) and their actions (e.g., driving), and provide grounding frames for involved entities. Compared with Action Recognition and Object Detection, GSR is more in line with human cognitive habits, better allowing law enforcement agencies to understand the predictions. However, the crucial issue with most existing frameworks is the neglect of verb ambiguity, that is, superficially similar verbs but have distinct meanings (e.g. buying v.s. giving). Many existing works propose a two-stage model, which first blindly predicts the verb, and then uses this verb information to predict semantic roles. These frameworks ignore the importance of noun information during verb prediction, making them susceptible to misidentifications. To address this problem and better discern between ambiguous verbs, we propose HiFormer, a novel hierarchical transformer framework with direct and comprehensive consideration of similar verbs for each image, to more accurately identify the salient verb, semantic roles, and the grounding frames. Compared with the state-of-the-art models in Grounded Situation Recognition (SituFormer and CoFormer), HiFormer shows an advantage of over 35% and 20% on the Top-1 and Top-5 verb accuracy respectively, as well as 13% on the Top-1 Noun accuracy.

A video monitoring and computational system for estimating migratory juvenile fish abundance in river systems

AbstractDiadromous fishes migrate between marine and fresh waters for reproduction. For anadromous species, which spawn in freshwater, improved access to freshwater spawning and nursery habitats and ability of juveniles to emigrate to the ocean may support population recovery. Despite the potentially enormous influence of early life stage survival on adult population size, managers and scientists have limited capacity to assess numbers of juvenile anadromous fishes leaving freshwater ecosystems. Such data are critical for evaluating reproductive success and habitat suitability and have been identified as a top priority in anadromous fish research and management. We developed a state‐of‐the‐art underwater video and computational system to collect videos to estimate abundances and migration timing for juvenile river herring (Alosa pseudoharengus; Alosa aestivalis). We collected continuous video in the Monument River (Bourne, Massachusetts, USA) from June to November 2017. We trained three types of neural network models to detect and count fish in video frames and evaluated model performance by comparing human counts to model outputs. Our top model assessed presence and absence (F1 = 87%) and counted fish (counting error 9.4%) with an accuracy comparable to human counters (F1 = 88%). Our system's capability to collect accurate counts of emigrating juveniles will provide critical information that could be related to the numbers of spawning adults, system‐specific productivity, and spawning and nursery habitat suitability. Both the video collection system and computational model may be transferrable to other sites and for other species where tracking juvenile emigration may inform management efforts.

Attention Mechanism with Spatial-Temporal Joint Deep Learning Model for the Forecasting of Short-Term Passenger Flow Distribution at the Railway Station

Accurate understanding of passenger flow distribution is crucial for effective station crowd management. However, due to the complexity and randomness of passenger flow and the unclear spatial-temporal correlation between functional areas within the station, predicting the spatiotemporal distribution dynamics of inflow and future short-term distribution trends is challenging. Emerging deep learning models offer valuable insights for accurately predicting passenger flow distribution. Thus, we propose a deep learning architecture, named “ST-Bi-LSTM,” which combines a bidirectional long short-term memory network with a spatial-temporal attention mechanism. Initially, we outline the methodologies of Bi-LSTM, the DeepWalk-based spatial attention mechanism, and the temporal attention mechanism. The spatial attention mechanism is employed to extract station spatial network topology information and enhance the representation of passenger flow characteristics in highly correlated areas during the forecasting process. Simultaneously, the temporal attention Bi-LSTM is utilized for capturing temporal correlations. The architecture comprises four branches dedicated to station real-time video monitoring data, spatial network topology, function area attributes, and train timetables. Subsequently, leveraging in-station CCTV data, passenger travel behavior data, and train timetables, we apply the architecture to the Tianjin West High-Speed Railway Station. We conduct a comparative analysis of the prediction performance and time complexity of the proposed architecture against existing baseline models, demonstrating superior performance and robustness exhibited by the ST-Bi-LSTM model (achieving a reduction in RMSE of over 10%). This study facilitates the transition of station management from passive response to active prediction of station passenger flow dynamics.

A Detecting Method of Users’ Reading State in Library via Modulated Light Illumination

ABSTRACT In order to overcome the shortcomings of the traditional methods for detecting the users’ reading state in a smart library building, a more effective detecting method of library users’ reading state is proposed. In this method, some posture characteristic parameters are put forward in order to represent the users’ reading state after analyzing the relationship between the users’ posture and their reading state. And then, one stripe light illuminating system of 3-D posture is designed by developing of a stripe projector and a video monitoring system, in which, the image of users’ posture with periodic stripe modulation is received on the video monitoring system. At last, two users’ posture stripe images are extracted by time sequence, one as a reference image and the other as a deformed image, and these images are superimposed by their pixel subtracting after extracting the stripes, so as to obtain a superimposed image that reflects the users’ posture variation characteristic. The users’ posture characteristic parameters can be gotten by calculating the area pixels and number of deformation regions of continuous pixels, and the changed pixels per unit time and per user based on the superimposed image. Therefore, the users’ reading state can be detected based on the relationship between the characteristic parameters and users’ reading state. This method can realize the accurate detecting of the reading users’ status and provide the basis to build the smart library.

An advanced multimodal driver-assistance prototype for emergency-vehicle detection

In the automotive industry, intelligent monitoring systems for advanced human-vehicle interaction aimed at enhancing the safety of drivers and passengers represent a rapidly growing area of research. Safe driving behavior relies on the driver’s awareness of the road context, enabling them to make appropriate decisions and act consistently in anomalous circumstances. A potentially dangerous situation can arise when an emergency vehicle rapidly approaches with sirens blaring. In such cases, it is crucial for the driver to perform the correct maneuvers to prioritize the emergency vehicle. For this purpose, an Advanced Driver Assistance System (ADAS) can provide timely alerts to the driver about an approaching emergency vehicle. In this work, we present a driver-assistance prototype that leverages multimodal information from an integrated audio and video monitoring system. In the initial stage, sound analysis technologies based on computational audio processing are employed to recognize the proximity of an emergency vehicle based on the sound of its siren. When such an event occurs, an in-vehicle monitoring system is activated, analyzing the driver’s facial patterns using deep-learning-based algorithms to assess their awareness. This work illustrates the design of such a prototype, presenting the hardware technologies, the software architecture, and the deep-learning algorithms for audio and video data analysis that make the driver-assistance prototype operational in a commercial car. At this initial experimental stage, the algorithms for analyzing the audio and video data have yielded promising results. The area under the precision-recall curve for siren identification stands at 0.92, while the accuracy in evaluating driver gaze orientation reaches 0.97. In conclusion, engaging in research within this field has the potential to significantly improve road safety by increasing driver awareness and facilitating timely and well-informed reactions to crucial situations. This could substantially reduce risks and ultimately protect lives on the road.

Identifying virtual care modality in electronic health record data.

Virtual care increased dramatically during the COVID-19 pandemic. The specific modality of virtual care (video, audio, eVisits, eConsults, and remote patient monitoring) has important implications for the accessibility and quality of care, but rates of use are relatively unknown. Methods for identifying virtual care modalities, especially in electronic health records (EHR) are inconsistent. This study (a) developed a method to identify virtual care modalities using EHR data and (b) described the distribution of these modalities over a 3-year study period. EHR data from 316 primary care safety net clinics throughout the study period (4/1/2020-3/31/2023) were included. Visit type (in-person vs virtual) by adults >18 years old were classified. Expert consultation informed the development of two algorithms to classify virtual care visit modalities; these algorithms prioritized different EHR data elements. We conducted descriptive analyses comparing algorithms and the frequency of virtual care modalities. Agreement between the algorithms was 96.5% for all visits and 89.3% for virtual care visits. The majority of disagreement between the algorithms was among encounters scheduled as audio-only but billed as a video visit. Restricting to visits where the algorithms agreed on visit modality, there were 2-fold more audio-only than video visits. Visit modality classification varies depending upon which data in the EHR are prioritized. Regardless of which algorithm is utilized, safety net clinics rely on audio-only and video visits to provide care in virtual visits. Elimination of reimbursement for audio visits may exacerbate existing inequities in care for low-income patients.

Unsupervised Conditional Diffusion Models in Video Anomaly Detection for Monitoring Dust Pollution.

Video surveillance is widely used in monitoring environmental pollution, particularly harmful dust. Currently, manual video monitoring remains the predominant method for analyzing potential pollution, which is inefficient and prone to errors. In this paper, we introduce a new unsupervised method based on latent diffusion models. Specifically, we propose a spatio-temporal network structure, which better integrates the spatial and temporal features of videos. Our conditional guidance mechanism samples frames of input videos to guide high-quality generation and obtains frame-level anomaly scores, comparing generated videos with original ones. We also propose an efficient compression strategy to reduce computational costs, allowing the model to perform in a latent space. The superiority of our method was demonstrated by numerical experiments in three public benchmarks and practical application analysis in coal mining over previous SOTA methods with better AUC, of at most over 3%. Our method accurately detects abnormal patterns in multiple challenging environmental monitoring scenarios, illustrating the potential application possibilities in the environmental protection domain and beyond.

Design and implementation of an automated baby monitoring cradle

AbstractEver increasing global birth rate, increased working women ratio and innovative technologies motivate the engineering society to introduce novel ideas for the baby caring. Considering this, we present hardware implementation of an automatic cradle for smart and remote monitoring. In contrast to previous implementations, we present the design insights such as swing angle, required force and torque, arc length and speed of the cradle for the first time in the literature. Furthermore, to make the cradle more comfortable and smooth swing, we introduce a novel Slider Crank Mechanism to control the cradle swing for the first time in the literature. This provides flexibility, and improved energy efficiency, while maintaining controlled and predictable reciprocating motion. The proposed solution is equipped with a number of technological features, that is, cry detection based swing, collision avoidance, WiFi based remote video monitoring, wet detection and baby health monitoring system.

The Impact of Adding a 2-Way Video Monitoring System on Falls and Costs for High-Risk Inpatients.

We aimed to investigate the value of adding a video monitoring (VM) system with falls and costs for patients at high risk. We conducted a retrospective, historically controlled study of adults (≥18 y old) at high risk of fall admitted at the University of Miami Hospital and Clinics from January 1 to November 30, 2020 (pre-VM) and January 1 to November 30, 2021 (post-VM); in-person sitters were available in both periods. Fall risk assessment was conducted on admission and at every nursing shift; we defined patients as high risk if their Morse Fall Scale was ≥60. We conducted a multivariable logistic regression model to evaluate the association of period (pre- versus post-VM) with falls and performed a cost analysis. Our primary cohort consisted of 9,034 patients at high risk of falls, 4,207 (46.6%) in the pre-VM and 4,827 (53.4%) in the post-VM period. Fall rates were higher in the pre- than the post-VM periods (3.5% versus 2.7%, P = 0.043). After adjustment, being admitted during the post-VM period was associated with a lower odds of fall (odds ratio [95% confidence interval], 0.49 [0.37-0.64], P < 0.001). The median adjusted hospital cost (in 2020 dollars) was $1,969 more for patients who fell than for patients who did not (interquartile range, $880-$2,273). Considering start-up and ongoing costs, we estimate VM implementation to partly replace in-person monitoring has potential annual cost savings of >$800,000 for a hospital similar to ours. Video monitoring to augment in-person sitters is an effective fall prevention initiative for patients at high risk of falls, which is likely also cost-effective.

MouseVUER: video based open-source system for laboratory mouse home-cage monitoring

Video monitoring of mice in the home-cage reveals behavior profiles without the disruptions caused by specialized test setups and makes it possible to quantify changes in behavior patterns continually over long time frames. Several commercial home-cage monitoring systems are available with varying costs and capabilities; however there are currently no open-source systems for home-cage monitoring. We present an open-source system for top-down video monitoring of research mice in a slightly modified home-cage. The system is designed for integration with Allentown NexGen ventilated racks and allows unobstructed view of up to three mice, but can also be operated outside the rack. The system has an easy to duplicate and assemble home-cage design along with a video acquisition solution. The system utilizes a depth video camera, and we demonstrate the robustness of depth video for home-cage mice monitoring. For researchers without access to Allentown NexGen ventilated racks, we provide designs and assembly instructions for a standalone non-ventilated rack solution that holds three systems for more compact and efficient housing. We make all the design files, along with detailed assembly and installation instructions, available on the project webpage (https://github.com/NIH-CIT-OIR-SPIS/MouseVUER).

Video Annomaly Classification Using Convolutional Neural Network

The use of surveillance videos is increasingly popular in city monitoring systems. Generally, the analysis process in surveillance videos still relies on conventional methods. This method requires professional personnel to constantly monitor and analyze videos to identify abnormal events. Consequently, the conventional approach is time-consuming, resource-intensive, and costly. Therefore, a system is needed to automatically detect video anomalies, reducing the massive human resource utilization for video monitoring. This research employs deep learning methods to classify anomalies in videos. The video anomaly detection process involves transforming the video into image format by extracting each frame present in the video. Subsequently, a Convolutional Neural Network (CNN) model is utilized to classify anomalous events within the video. Testing results using the CNN architectures DenseNet121 and EfficientNet V2 yielded performance accuracies of 87% and 75%, respectively. The testing results indicate that the DenseNet121 architecture outperforms the EfficientNetV2 architecture in terms of performance.

Choosing optimal distance between ground-based technical observation posts

The article develops an approach to rational video monitoring construction of spatially distributed objects territories. Target. Justification for choice optimal distance between ground technical observation posts (TOP) based on electro-optical surveillance equipment in the visible and thermal imaging ranges. Results. The developed model makes it possible to determine maximum distance between adjacent TOPs, ensuring guaranteed surface targets detection and recognition in standard conditions. Practical significance. The optimal distance between TOPs ensures not only guaranteed quality of monitoring of territory boundaries, but also minimum linear cost of the security line.

An improved SSD lightweight network with coordinate attention for aircraft target recognition in scene videos

Accurate identification and monitoring of aircraft on the airport surface can assist managers in rational scheduling and reduce the probability of aircraft conflicts, an important application value for constructing a "smart airport." For the airport surface video monitoring, there are small aircraft targets, aircraft obscuring each other, and affected by different weather, the aircraft target clarity is low, and other complex monitoring problems. In this paper, a lightweight model network for video aircraft recognition in airport field video in complex environments is proposed based on SSD network incorporating coordinate attention mechanism. First, the model designs a lightweight feature extraction network with five feature extraction layers. Each feature extraction layer consists of two modules, Block_A and Block_I. The Block_A module incorporates the coordinate attention mechanism and the channel attention mechanism to improve the detection of obscured aircraft and to enhance the detection of small targets. The Block_I module uses multi-scale feature fusion to extract feature information with rich semantic meaning to enhance the feature extraction capability of the network in complex environments. Then, the designed feature extraction network is applied to the improved SSD detection algorithm, which enhances the recognition accuracy of airport field aircraft in complex environments. It was tested and subjected to ablation experiments under different complex weather conditions. The results show that compared with the Faster R-CNN, SSD, and YOLOv3 models, the detection accuracy of the improved model has been increased by 3.2%, 14.3%, and 10.9%, respectively, and the model parameters have been reduced by 83.9%, 73.1%, and 78.2% respectively. Compared with the YOLOv5 model, the model parameters are reduced by 38.9% when the detection accuracy is close, and the detection speed is increased by 24.4%, reaching 38.2fps, which can well meet the demand for real-time detection of aircraft on airport surfaces.

SERUM FENTANYL CONCENTRATIONS AND BEHAVIOR ASSOCIATED WITH TRANSDERMAL FENTANYL APPLICATION ON HEALTHY CORN SNAKES (PANTHEROPHIS GUTTATUS).

Snakes are common household pets and frequently managed in zoos. Geriatric snakes commonly develop osteoarthritis, leading to a declining quality of life that often results in euthanasia. Anecdotally, the application of transdermal fentanyl patches (TFP) appears to contribute to clinical improvement, including increased activity level, in osteoarthritic snakes presumed to be in pain. This study evaluated serum fentanyl concentrations over time and the effects of TFP on the normal behavior of healthy, captive, adult corn snakes (Pantherophis guttatus) using constant video monitoring. Serum fentanyl concentrations were evaluated over 4 wk during 12.5 µg/h TFP application, and the results demonstrated long-lasting (>4 wk) serum concentrations that were consistent with analgesic efficacy in mammalian species during TFP application. At 4 wk of TFP application, mean serum fentanyl concentrations were 11.5 ± 5.5 ng/ml. Snakes were videotaped for 1 wk prior to and 2 wk after 12.5 µg/h TFP application, and behavior was evaluated by an ethogram. Behavioral changes associated with TFP application included decreased mean time spent active, decreased mean number of climbs, and decreased mean number of water visits; feeding behavior was unchanged. Overall, these results suggest that TFP application may provide safe, clinically effective analgesia in healthy corn snakes for at least 4 wk without inducing deleterious side effects, and may therefore be appropriate analgesia for management of osteoarthritic snakes.

Defect monitoring in additive manufacturing through OpenCV-based remote video access

Errors in additive manufacturing are a significant obstacle to its widespread adoption in various application. The developments of 3D printed parts are prone to various types of defects, which are difficult to monitor in real-time through human vision. The use of 3D printers has increased in diverse fields, leading to a significant rise in the need of continuous monitoring during the manufacturing process. Prompt action should be taken to stop or correct errors immediately to prevent a defective part. However, continuous monitoring of 3D printer through human vision is impractical for longer tasks and cannot be accurate. Human inspection is not infallible and could introduce its own errors. Given the multitude of application, continuous defect monitoring is extremely challenging, particularly for longer tasks. Based on these constraints a suitable remote monitoring system through open-source computer vision has been proposed. Incorporating the concept of Internet of things, the video of printing process could be viewed by a user or controller from a remote location enabling to monitor the printing process. With the use of Haar classifier in the program, the printing defects that occur in the 3D printed part could be identified immediately through contours drawn over the defective part. Depending upon the severity of defect indicated, the user could take respective action to control the defect formation. With Industry 4.0 in the implementation stages, this method of video monitoring of defects in 3D printers could be an effective method for widespread use of additive manufacturing systems in industries.

АКТУАЛЬНОСТЬ ПРОВЕДЕНИЯ МОНИТОРИНГА СЕЛЬСКОХОЗЯЙСТВЕННЫХ КУЛЬТУР И ХВОЙНОГО ПОДРОСТА В ВЕГЕТАТИВНЫЙ ПЕРИОД

Modern agriculture faces a number of challenges, including climate change, resource constraints and the need to increase productivity. In this context, the assessment of quantitative parameters of crop growth by technical means during the growing season becomes particularly relevant. The scientific approach to this problem is becoming a key tool for optimizing production and ensuring food security, sustainable development of forest ecosystems. (Research purpose) To substantiate the relevance of monitoring crops and undergrowth during artificial reforestation during the vegetative period. (Materials and methods) Video monitoring of agricultural crops, automated segment recognition, geospatial data, literature review, statistical analysis were used. (Results and discussion) The statistics and problems of existing measures for monitoring coniferous undergrowth after reforestation measures were analyzed. The methods of using video monitoring of agricultural crops, in particular, for analyzing the yield of corn shoots, were considered. The use of a video monitoring system to control the care of coniferous undergrowth in the first five years after reforestation was substantiated. Key limitations and difficulties were cited, including high labor costs, insufficient monitoring, lack of systematic approach, limited human labor and low speed and accuracy of data processing. It has been shown that these factors emphasize the need for innovations in methods of controlling the growth of crops and caring for coniferous young. (Conclusions) It was revealed that the video monitoring system of coniferous undergrowth is one of the solutions to the problems of monitoring the condition of forest areas. It was determined that it allows assessing the quality of planting work, determining the growth and health indicators of seedlings, monitoring the environment by indirect signs and, due to a high level of digitalization and automation, responding quickly to violations. It was stated that the assessment of the parameters of crop growth in real time with the use of technical solutions makes it possible to identify problems such as diseases and parasites at an early stage, which will improve the possibilities for their prevention.

Social effects on behaviorally-scored and pedometer-detected estrus in beef cattle.

The number of cows in estrus often influences estrus behavior; however, the effects of social order are not well documented. This study examined the effects of social order on the expression of behaviorally-scored and pedometer-detected estrus, combined with the effects of the number of cows in estrus. In a herd comprising 13 or 15 beef cattle, cows with orders 1st-7th were defined as dominant and the remaining cows as subordinate. Sole or simultaneous estrus was induced by prostaglandin F2α analog injection and/or intravaginal progesterone treatment. Ovulation timing was determined using ultrasonography at 6-hour intervals. Estrous signs and steps of the cows were recorded 49 h before ovulation using video monitoring and a pedometer, respectively. Among the 59 treated cows, 56 behaviorally-scored estruses (27 sole and 29 simultaneous) were detected. In the sole estrus, 61.5% of the dominant-rank cows had no zero-point period; however, 35.7% of the subordinate-rank cows had that period. The dominant-rank cows in estrus alone had a significantly shorter duration of scored estrus than those in simultaneous estrus (P < 0.05). Among the 50 pedometer-detected estruses (24 sole and 26 simultaneous), the subordinate-rank cows in sole estrus had a shorter interval from estrus onset to ovulation than the dominant-rank cows in simultaneous estrus (P < 0.05). The effects of social order varied in response to the number of cows in estrus, which might have influenced determining the optimal time for artificial insemination.

Embryonic development of the scorpion mud turtle (Kinosternon scorpioides) bred in captivity.

The developmental biology of Kinosternon scorpioides is described, based on the phenotype. This species is important for the flora because they are excellent seed disseminators. In addition, basic embryological information is not yet fully clarified, and this research provides unprecedented information on the chelonian embryology of the Amazonian fauna. The present study aims to identify the embryology of K. scorpioides in captivity during different periods. Females were monitored throughout the reproductive cycle, by video monitoring, to identify nests and the presence of newly laid eggs. At regular weekly intervals, embryo samples were collected fixed in a 4% paraformol solution and preserved in 70% alcohol. For the embryonic characterization, we used a stereomicroscope and the scanning electron microscopy method. We describe 15 embryonic stages for a 15-week (105-day) incubation process. Only at 42 days (6th week) was the morphological characterization of a chelonian observed and at the 12th week (Stage XII), the phenotypic characterization of the species K. scorpioides. In view of the evidence, we found that these phases are similar to the other turtles, with structural variations in the appearance and disappearance of structures due to the specific characteristics of the species.

Image-Based Process Monitoring using Projective Nonnegative-Tensor Factorization

In this study, a novel process monitoring method using image data is presented. The new method is referred to as projective nonnegative-tensor factorization (PNTF). Video monitoring systems are increasingly being deployed in complex industrial processes such as manu-facturing processes, food industries, and pharmaceutical process, for extracting information from images to achieve the purpose of process monitoring. Traditional process-monitoring techniques fail to take full advantage of the rich information from color images due to the complex data characteristics including the high-dimensionality and correlation structures. The proposed PNTF is capable of modeling the high-dimensionality structure of red-green-blue data of color images using a tensor analysis technique. Moreover, in contrast to the traditional data-processing algorithms such as principal component analysis, linear discriminant analysis and so on, the PNTF is more applicable to analyzing image data for its nonnegative constraints on matrix factors. In addition, the PNTF allows us to extract features from both grayscale and color images, making process monitoring using multivariate control charts achiev- able. At last, based on the PNTF, methods for detecting and isolating process faults are developed and evaluated through a case study on a production process of electrically melting magnesia.

Improving the organization of communication in the forest fire monitoring system

The article discusses the issues of early detection of forest fires. Currently, traditional monitoring methods are gradually being replaced by modern technologies based on automation and the use of artificial intelligence. The system of remote video monitoring “Lesohranitel”, which has been widely used in many regions of Russia, is considered in detail. The main advantages of this system and the basic principle of operation are considered. The scheme of communication organization of the considered system is presented, the main ways for its improvement are defined.