Edge Camera Research Articles

STDF: Spatio-Temporal Deformable Fusion for Video Quality Enhancement on Embedded Platforms

With the development of embedded systems and deep learning, it is feasible to combine them for offering various and convenient human-centered services, which is based on high-quality (HQ) videos. However, due to the limit of video traffic load and unavoidable noise, the visual quality of an image from an edge camera may degrade significantly, influencing the overall video and service quality. To maintain video stability, video quality enhancement (QE), aiming at recovering HQ videos from their distorted low-quality (LQ) sources, has aroused increasing attention in recent years. The key challenge for video QE lies in how to effectively aggregate complementary information from multiple frames (i.e., temporal fusion). To handle diverse motion in videos, existing methods commonly apply motion compensation before the temporal fusion. However, the motion field estimated from the distorted LQ video tends to be inaccurate and unreliable, thereby resulting in ineffective fusion and restoration. In addition, motion estimation for consecutive frames is generally conducted in a pairwise manner, which leads to expensive and inefficient computation. In this article, we propose a fast yet effective temporal fusion scheme for video QE by incorporating a novel Spatio-Temporal Deformable Convolution (STDC) to simultaneously compensate motion and aggregate temporal information. Specifically, the proposed temporal fusion scheme takes a target frame along with its adjacent reference frames as input to jointly estimate an offset field to deform the spatio-temporal sampling positions of convolution. As a result, complementary information from multiple frames can be fused within the STDC operation in one forward pass. Extensive experimental results on three benchmark datasets show that our method performs favorably to the state of the art in terms of accuracy and efficiency.

Effect of stacking sequence and interlayer toughening in ＋/−45 laminates

This paper investigates the combined effects of stacking sequence and interlayer toughening on the mechanical response and failure modes of ±45° carbon fiber reinforced laminates, comprising experimental and numerical studies. Two material systems (T800H/3631, T800S/3900) and three stacking sequences (4 double plies, 4 plies, 8 plies) have been analyzed. Digital Image Correlation (DIC) and a high resolution edge camera were utilized to capture the intricate crack patterns in each ply. This paper also addresses the challenge of the nonlinear in-plane shear characterization of the lamina, and proposes an amendment to the existing ASTM standard to obtain the in-situ non-linear response and the lamina shear strength with the consideration of residual stresses. The complex progression of micro-damage, matrix cracks and delamination as well as the final two-piece failure were captured by the enhanced semi-discrete damage modeling (eSD2M) framework. A parametric study was performed to reverse-engineer the most critical parameters such as interlayer strengths and toughnesses. The obtained values reflect the difference in the interlayer composition between the two material systems.

A Coalition Formation Approach for Privacy and Energy-Aware Split Deep Learning Inference in Edge Camera Network

Recently, an increasing amount of application tasks have depended on deep learning (DL) inference models on Internet of Things (IoT) based camera networks. However, it is challenging to perform the inference of such resource-hungry DL models on the computationally limited IoT system. Compared to cloud computing, edge computing deploys resources near the end-users to reduce the transmission delay and retains the raw data on the trusted servers to mitigate privacy concerns. Since resources at the edge are limited, management like user association decisions, edge resources allocation, and device configuration with DL model parameter selection becomes essential. This work proposes a coalition formation game-based algorithm to solve the association problem between IoT-based cameras and the edge nodes. Our goal is to maximize the social welfare that consists of multi-view detection enhancement, the privacy retained preference, and the power savings from the cameras. Besides, we adopt the concept of split-ML to provide more flexibility for networking and computing resources allocation at the edge. The final coalition structure is proved to converge and maintain stability. The simulation results show that each design knob, including association decisions made by coalition formation, DNN layer-level partition, and multi-view detection, is essential under different scenario settings.

Dynamic Offloading on a Hybrid Edge–Cloud Architecture for Multiobject Tracking

Multiobject tracking (MOT) using computer vision on smart cameras has become more popular owing to its continuously improving performance. However, the number of tracking targets may suddenly increase, which deteriorates the accuracy and robustness of MOT on existing edge cameras (smart cameras leveraging edge intelligence) because of their restricted computing and storage capacities. To address this issue, we propose dynamic computation offloading on a hybrid edge–cloud architecture. When the confidence of MOT decreases because of a sudden increase in tracking targets, an edge camera can dynamically offload its MOT tasks to backend servers and retake the tasks once confidence is recovered. The experiment results show that the accuracy of the hybrid system can be increased by up to 8% while reducing the network traffic. Therefore, the proposed hybrid approaches can maintain not only the quality and efficiency of image processing on edge cameras but also good tracking performance while reducing communication costs in an environment with dynamically changing people flow.

Lightweight and Dynamic Deblurring for IoT-Enabled Smart Cameras

Blurred images often significantly influence the stability of computer vision systems. With the development of the Internet of Things (IoT), a camera that leverages artificial intelligence and edge computing (hereafter referred to as an edge camera) can enhance the robustness of an IoT service. Although previous studies have used deep neural networks (DNNs) for image deblurring, existing image-deblurring techniques result in over-smoothing, even with a lightweight model design. In addition, previous deblurring studies have not considered the quality of the input image first; thus, a clear input image would be processed with unnecessary deblurring. Therefore, our study proposes a lightweight and dynamic border-enhancement deblurring deep network that can be operated with high performance on edge cameras, and the network can assess the quality of input images to determine whether deblurring is necessary. The experimental results show that our lightweight deblurring approach outperforms existing studies by up to 8%, given that the speed also improved by 33.78%. As for the dynamic deblurring approach, our results show that the image metrics can be slightly improved and that speed is improved by 294.5%.

Goal and Elite-Data-Driven Anthropomorphic Learning for Streaming-Image Analytics

With the rise of the Internet of Things (IoT), many edge cameras (smart cameras that leverage edge computing) have provided real-time streaming-image analytics using anthropomorphic learning. However, previous anthropomorphic-learning studies did not allow the setting of customized goals to filter out concerned features during online model learning. In addition, when any concept drift occurs, the existing studies using an edge camera cannot adapt efficiently using only representative images. Our study therefore proposes solutions to the above issues based on the self-regulated learning framework, which divides a learning procedure into before, during, and after stages to improve overall learning performance, just like human beings. In the before learning stage of a model on an edge camera, we propose a generative adversarial network (GAN)-based customized feature filtering to screen out users’ concerned features before real-time model learning of image analytics. Next, during the learning stage of the model, we also propose keyframe-oriented model adaptation that can effectively respond to concept drift while reducing image storage space on an edge camera. The experimental results show that customized concerned feature filtering can improve the adversarial effect of the concerned features by 24.26 percent on average, meaning it can avoid using unwanted features from learning an undesirable model. The keyframe-oriented model adaptation can accelerate the overall adjustment time by 52 percent and save the required storage space by 54.7 percent at the cost of compromising precision by 2 percent at most.

Direct Edge-to-Edge Many-to-Many Latent Feature Transfer Learning

Smart cameras that leverage edge-computing (edge cameras) are increasingly being combined with deep neural networks to realize the Artificial Intelligence of Things (AIoT), enabling a smart life with “low-touch services,” such as unmanned stores. However, deploying many edge cameras and training their models (edge models) in unmanned stores is time consuming and labor intensive. Therefore, studies have utilized transfer learning methods, but training edge models often need powerful servers. Although we previously proposed direct edge-to-edge instance transfer, this approach did not exploit latent features, still required high bandwidth and long training time, and caused privacy leakage. Therefore, we propose direct edge-to-edge many-to-many latent feature transfer learning, which includes elite latent feature (ELF) extraction, direct edge-to-edge one-to-many latent feature transfer learning (DeOmf), and direct edge-to-edge many-to-one latent feature transfer learning (DeMof). Through ELF extraction, DeOmf allows one source edge camera to transfer latent features to multiple target edge cameras, which improves knowledge reuse and accelerates initial model training. Further, DeMof can help exploit the diversity of multiple source edge cameras for one target edge camera. The experimental results show that DeOmf improves accuracy by 6.30%, reduces training time by 32.15%, and saves 22.92% (maximum 83.33%) on transmission cost. In addition, DeMof increases accuracy by 3.42% and saves 66.99% on training time and 56.67% on transmission cost. These improvements can greatly broaden the applicability of our proposed system when comparing to instance-based transfer learning.

Environment-Aware Dense Video Captioning for IoT-Enabled Edge Cameras

In recent years, the Artificial Intelligence of Things (AIoT) has led to the rapid development of edge computing, and existing video-captioning systems can be deployed directly on AIoT-enabled cameras (hereafter referred to as edge cameras), which have increasingly powerful computing resources. Therefore, we propose a lightweight dense-video-captioning model based on the Transformer framework to improve execution efficiency for video-caption generation on edge cameras. In addition, to investigate the effect of concept drift on video captioning, we also propose an environment-aware adaptation to allow the system to respond to changes in the environment in order to produce more accurate video captions. Experimental results on the lightweight dense-video-captioning model show that the bilingual evaluation understudy (BLEU) metrics can increase by up to 23.5%, the computation time decreases by 46.4%, and the edge camera operates at a rate of 27.63 FPS, which is also faster than existing approaches by 4.7%. In addition, the mean average precision of the environment-aware adaptation is up to 11.3% higher than the existing approaches. In conclusion, the proposed approaches perform better than previous ones and are more flexible in different weather scenarios.

Lights, camera, operate!

Our Techonologist looks at cutting edge camera technology being used to operate by surgeons.

Estimating Interpersonal Distance and Crowd Density with a Single-Edge Camera

For public safety and physical security, currently more than a billion closed-circuit television (CCTV) cameras are in use around the world. Proliferation of artificial intelligence (AI) and machine/deep learning (M/DL) technologies have gained significant applications including crowd surveillance. The state-of-the-art distance and area estimation algorithms either need multiple cameras or a reference object as a ground truth. It is an open question to obtain an estimation using a single camera without a scale reference. In this paper, we propose a novel solution called E-SEC, which estimates interpersonal distance between a pair of dynamic human objects, area occupied by a dynamic crowd, and density using a single edge camera. The E-SEC framework comprises edge CCTV cameras responsible for capturing a crowd on video frames leveraging a customized YOLOv3 model for human detection. E-SEC contributes an interpersonal distance estimation algorithm vital for monitoring the social distancing of a crowd, and an area estimation algorithm for dynamically determining an area occupied by a crowd with changing size and position. A unified output module generates the crowd size, interpersonal distances, social distancing violations, area, and density per every frame. Experimental results validate the accuracy and efficiency of E-SEC with a range of different video datasets.

엣지카메라기반 실시간 도로위험 검지 및 정보 제공 시스템

현재 C-ITS 서비스는 레이다 혹 카메라 기반으로 돌발상황을 검지하고 도로위험 정보를 운전자에게 제공한다. 레이다기반의 돌발상황검지기는 가격이 비싸고 교통 이상 상황을 확인하기 위해서는 별도의 카메라가 필요하다. 또한, 레이다는 차량과 보행자 검출이 가능하나 그 외 오토바이, 자전거, 낙하물과 같은 객체는 검출이 어렵다. 기존 C-ITS 시스템에서 카메라기반의 돌방상황검지의 경우, C-ITS 센터의 영상분석 서버에서 돌발상황을 검지하고 위험정보를 운전자에게 제공하기 때문에 실시간 서비스를 제공하기 어렵다. 이러한 문제를 해결하기 위해 본 논문은 엣지카메라기반 실시간 도로위험인식 및 정보제공시스템을 제안한다. 엣지카메라는 로컬에서 영상처리를 통해 객체를 인식하고 위험상황을 판단하여 RSU를 통해 운전자에게 위험정보를 제공한다. 제안 시스템은 실시간 처리를 위해 경량 딥러닝 네트워크를 사용하고, 정확도 향상을 위해 지역최적화를 수행한다. 그리고 기존 C-ITS 응용 표준 프로토콜을 수정 없이 사용할 수 있도록 엣지카메라와 RSU의 통신 인터페이스를 설계하였다. 실험을 통해 교통객체를 정확하게 인식하고 100ms 이내에 실시간으로 도로위험상황을 운전자에게 정보를 제공함을 확인할 수 있다.

An Analysis of the Optical System of a Length Measurement Comparator

In the paper, the optical system of a precise bar length measuring comparator is analysed. The requirements for such a system are determined and systematized. The impact of the resolution, signal discretization frequency, image blurring, bar edge measurement indeterminacy and camera noise on measuring the bar width and establishing the bar position is discussed upon. Various algorithms have been analysed and finally a bar edge identification algorithm oriented to the scale calibration accuracy and the high processing speed was proposed. In the end of the paper, conclusions are provided.

Calibration of Satellite Imagery with Multispectral UAV Imagery

Unmanned aerial vehicle (UAV)-based multispectral remote sensing has shown a tremendous potential normalized difference vegetation index (NDVI) for precision agriculture. In this study, data captured from a UAV equipped with a Multispectral Mica Sense Red Edge camera used as ground-truth information to calibrate Sentinel-2 imagery. UAV-based NDVI allowed crop estimation at 10-cm pixel resolution by discriminating no-green vegetation pixels. The reflectance value and NDVI of the crops at different stages were derived from both UAV and Sentinel-2 images. The UAV Multispectral mapping method used in this study provided advanced information about the physical conditions of the study area (Roorkee) and improved land feature delineation. The result shows that UAV data produced more accurate reflectance values than Sentinel-2 imagery. However, the accuracy of the vegetation index is not wholly dependent on the accuracy of the reflectance. The UAV-derived NDVI has relatively low sensitivity to the vegetation coverage and insignificantly affected by environmental factors compared to NDVI derived from Sentinel-2 image.

Toward Direct Edge-to-Edge Transfer Learning for IoT-Enabled Edge Cameras

Due to advances in edge computing, image recognition via smart cameras (hereafter referred to as edge cameras) has facilitated the development of unmanned stores. However, it is very time consuming and costly to collect labeled data for training initial models for an edge camera. Although existing transfer learning can speed up model training, it must depend on a powerful centralized server with considerable human intervention, thus hindering the development of autonomous and collaborative edge learning. To address this issue, our study proposes direct edge-to-edge (e2e) collaborative transfer learning with three key technologies. The first is elite-instance-based matching to utilize and transmit only representative images for transfer learning to decrease network cost among edge cameras. The second is one-to-many e2e transfer learning, which can increase the knowledge reusability of a single source camera to build multiple target models. The last is many-to-one e2e transfer learning, which enables a target edge camera to reuse knowledge from multiple sources to further decrease the effort in labeled data collection. The experimental results show that elite-instance-based matching can effectively save up to 70% of source samples on average that need to be transmitted for initial model training, and it further improves the accuracy of existing one-to-many and many-to-one e2e transfer learning.

Environment-Aware Multiscene Image Enhancement for Internet of Things Enabled Edge Cameras

With the development of the Internet of Things, it is feasible to leverage as much computing resources on an IoT-enabled camera to realize edge intelligence (hereafter referred to as an edge camera) for providing various human-centered services. However, environmental interference can significantly degrade the visual quality of an image into edge cameras, thereby affecting the overall image and service quality. In order to maintain image stability, image enhancement using complex deep networks has been widely utilized. As edge cameras often have restricted computing power and cannot run complex neural networks efficiently, we design a lightweight residual enhancement network, which incorporates highly efficient convolution operations as basic building blocks. With this modular design on a generative adversarial network training framework, the resultant network can be easily optimized according to hardware specifications of a given edge camera. To accommodate various weather conditions for an edge camera, we further design an environment-aware model deployer that can detect the weather conditions and deploy the most suitable image-enhancement model. The experimental results show that the lightweight residual enhancement network can produce image quality close to what past research demonstrated while increasing running speeds by more than 30%. In a streaming video to simulate changing weather, the results also demonstrate the effectiveness of the proposed environment-aware model deployer to maintain the quality of the images under a dynamic environment.

Three-dimensional mechanical parts reconstruction technology based on two-dimensional image

With the development of computer technology and three-dimensional reconstruction technology, three-dimensional reconstruction based on visual images has become one of the research hotspots in computer graphics. Three-dimensional reconstruction based on visual image can be divided into three-dimensional reconstruction based on single photo and video. As an indirect three-dimensional modeling technology, this method is widely used in the fields of film and television production, cultural relics restoration, mechanical manufacturing, and medical health. This article studies and designs a stereo vision system based on two-dimensional image modeling technology. The system can be divided into image processing, camera calibration, stereo matching, three-dimensional point reconstruction, and model reconstruction. In the part of image processing, common image processing methods, feature point extraction algorithm, and edge extraction algorithm are studied. On this basis, interactive local corner extraction algorithm and interactive local edge detection algorithm are proposed. It is found that the Harris algorithm can effectively remove the features of less information and easy to generate clustering phenomenon. At the same time, the method of limit constraints is used to match the feature points extracted from the image. This method has high matching accuracy and short time. The experimental research has achieved good matching results. Using the platform of binocular stereo vision system, each step in the process of three-dimensional reconstruction has achieved high accuracy, thus achieving the three-dimensional reconstruction of the target object. Finally, based on the research of three-dimensional reconstruction of mechanical parts and the designed binocular stereo vision system platform, the experimental results of edge detection, camera calibration, stereo matching, and three-dimensional model reconstruction in the process of three-dimensional reconstruction are obtained, and the full text is summarized, analyzed, and prospected.

VU: Edge Computing-Enabled Video Usefulness Detection and its Application in Large-Scale Video Surveillance Systems

In the era of smart and connected communities, video surveillance systems, which typically involve tens to thousands of cameras, have increasingly become prominent components for public safety. In current practice, when a failure occurs in a video surveillance system, the operation and maintenance teams usually spend a substantial amount of time locating and identifying the failure; hence, the fast online response cannot be guaranteed in a large-scale video surveillance system. Meanwhile, the video data that contains potential failures consumes bandwidth that could be used for useful video data. The useless video will waste the scarce bandwidth in the network and storage usage in the cloud. The emergence of edge computing is highly promising in video preprocessing with an edge camera. A video surveillance system is a killer application for edge computing. In this article, we propose an edge computing-enabled video usefulness (i.e., VU) model for large-scale video surveillance systems. We also explore its application, e.g., early failure detection and bandwidth improvement. According to the usefulness of the video data, the VU model can locate a failure and send it to end-users on the fly. In this article, our goals are threefold: 1) proposing a comprehensive VU model. To the best of our knowledge, this is the first work to explore the feasibility of the VU model and to determine VU values in a real application; 2) reducing the mean time to detection (i.e., MTTD) efficiently via edge computing-enabled fast online failure detection approaches; and 3) relieving the network bandwidth for large-scale video surveillance systems. Our experimental results demonstrate the approaches in VU model accurately detect failures that were collected from a video surveillance system with approximately 4000 cameras. The MTTD is substantially shortened by the fast online detection approaches. The video data with the worst VU values is mostly discarded to lessen overload of the network.

Progressive matrix cracking in carbon/epoxy cross-ply laminates under static and fatigue loading

This paper presents an experimental investigation into the growth of matrix cracks in carbon/epoxy cross-ply laminates under static and fatigue loading. Particular emphasis is placed on how to address fatigue damage. Two models are proposed to describe the evolution of fatigue cracks, where the crack density is used as the mechanistic damage variable. The first model defines the evolution of cracks according to a Paris law function. The second assumes that fatigue degradation is mainly caused by diffuse damage, resulting in an apparent decrease in the fracture energy, so that the cracking process is controlled by the static function. Both models are analysed in detail and compared against experimental data for different transverse plies and loading levels. The presence of cracks is monitored by enhanced X-ray radiography and edge camera images. Two crack density master curves are derived, one based on fracture mechanics for static and one based on a Paris law for fatigue cracking. The predictions show good correlation with the experiments.

A divertor scraper observation system for the Wendelstein 7-X stellarator

Two graphite divertor elements called scrapers have been installed on the Wendelstein 7-X stellarator in the throat of the magnetic island divertor. To diagnose one, we have designed, built, calibrated, and installed a new infrared/visible imaging endoscope system to enable detailed observations of the plasma interactions and heat loads at one of the scrapers and the nearby divertor surfaces. The new system uses a shuttered pinhole-protected pair of 90° off-axis 228 mm focal length aluminum parabolic mirrors, and two flat turning metal mirrors, to send light to a sapphire vacuum window 1.6 meters away, beyond which we have co-located telephoto lens-based infrared and visible cameras. The back-to-back off-axis parabolas serve to cancel out most aberrations, enabling the use of off-the-shelf commercial optics outside of the vessel. For the infrared, we use a 3-5 μm 1-megapixel FLIR SC8303HD camera and for the visible, a 5-megapixel CMOS PCO 5.5 edge camera. A short 1-m quartz pickoff fiber is used to send 200-1100 nm light to a compact spectrometer, also located in the same iron shield box as the cameras. The camera field of view covers the 700 mm length of the scraper, and includes locations monitored by thermocouples and Langmuir probes embedded in some of the scraper tiles. Predicted and actual optical test performances of the overall system are compared.

ROLE OF TIE-POINTS DISTRIBUTION IN AERIAL PHOTOGRAPHY

Abstract. Automatic image matching algorithms, and especially feature-based methods, profoundly changed our understanding and requirements of tie points. The number of tie points has increased by orders of magnitude, yet the notions of accuracy and reliability of tie points remain equally important. The spatial distribution of tie points is less predictable, and is subject only to limited control. Feature-based methods also highlighted a conceptual division of the matching process into two separate stages – feature extraction and feature matching. In this paper we discuss whether spatial distribution requirements, such as Von Gruber positions, are still relevant to modern matching methods. We argue that forcing such patterns might no longer be required in the feature extraction stage. However, we claim spatial distribution is important in the feature matching stage. We will focus on terrains that are notorious for difficult matching, such as water bodies, with real data obtained by users of VisionMap’s A3 Edge camera and LightSpeed photogrammetric suite.