Smart City Surveillance Research Articles

Editor’s Note: In Defense of Freedoms: Decentralized Networks as a Shield Against AI-Powered State Surveillance in Smart Cities

Associate Editor Ahmed Alrawi discusses the advancement of AI-powered state surveillance in smart cities and how they have sparked concerns. Decentralized systems may present a solution. Dispersed dat and systems make it more difficult for government security agencies and urban administrators to conduct widespread surveillance but also empower individuals with greater control over their personal data.

Two-timescale joint service caching and resource allocation for task offloading with edge–cloud cooperation

Task offloading with edge–cloud cooperation has emerged as a pivotal solution for meeting the intricate array of application coupled with dynamically evolving business demand in 6G business scenarios, such as traffic sensing, environmental monitoring, and video surveillance in smart cities. Nonetheless, effectively leveraging heterogeneous edge–cloud network resources for effective task offloading presents substantial challenges. Additionally, the inherent differences in system decision cycles escalate the complexity of the task offloading problem to a new dimension. In this study, we delve into a two-timescale joint service caching and resource allocation optimization for task offloading within edge–cloud cooperation aiming to maximize long-term network performance while adhering to energy constraints. We propose a novel edge–cloud cooperation task offloading scheme that supports both edge–cloud and edge–edge cooperation to effectively balance the edge–cloud and edge–edge loads, promoting the efficient co-utilization of all edge–cloud system resources. Furthermore, we devise an online two-timescale Lyapunov-based joint optimization framework for service caching, task offloading, and computing resource allocation. Our two-timescale decision-making framework can flexibly accommodate the inherent differences in the sensitive decision optimization periods, thereby mitigating the degradation of task offloading performance caused by frequent service caching updates. Finally, theoretical analysis confirms that our proposed algorithm can converge to an approximate optimal solution in polynomial time, and the superiority of our scheme is validated by extensive simulation experiments.

An Adversarial Machine Learning Based Approach for Privacy Preserving Face Recognition in Distributed Smart City Surveillance

Smart cities rely heavily on surveillance cameras for urban management and security. However, the extensive use of these cameras also raises significant concerns regarding data privacy. Unauthorized access to facial data captured by these cameras and the potential for misuse of this data poses serious threats to individuals’ privacy. Current privacy preservation solutions often compromise data usability with noise application-based approaches and vulnerable centralized data handling settings. To address these privacy challenges, we propose a novel approach that combines Adversarial Machine Learning (AML) with Federated Learning (FL). Our approach involves the use of a noise generator that perturbs surveillance data right from the source before they leave the surveillance cameras. By exclusively training the Federated Learning model on these perturbed samples, we ensure that sensitive biometric features are not shared with centralized servers. Instead, such data remains on local devices (e.g., cameras), thereby ensuring that data privacy is maintained. We performed a thorough real-world evaluation of the proposed method and achieved an accuracy of around 99.95% in standard machine learning settings. In distributed settings, we achieved an accuracy of around 96.24% using federated learning, demonstrating the practicality and effectiveness of the proposed solution.11The code is available at: https://github.com/farah-wahida/Privacy-Preserving-Face-Recognition-in-Distributed-Smart-City-Surveillance.

Efficiently adapting large pre-trained models for real-time violence recognition in smart city surveillance

Efficiently adapting large pre-trained models for real-time violence recognition in smart city surveillance

Smart surveillance using IoT: a review

In today’s modern society, video surveillance is a growing trend and it can revolutionize many aspects of technology, especially in future smart cities that will transform traditional surveillance systems into intelligent and interconnected networks. It may be difficult for even well-trained employees to process and respond immediately to monitored data. Moreover, IoT-enabled surveillance systems overcome the challenges and flaws of conventional passive monitoring techniques by offering real-time surveillance and automated notifications for suspicious activity, intrusions, or anomalies. Therefore, the objective of this article is to provide an Internet of Things (IoT)-based smart surveillance system that may successfully connect the ecosystem, resulting in enhanced monitoring for smart city services. The primary goal of this study is to provide a comprehensive review of several IoT-based surveillance techniques used in various smart city applications. The categorization of tasks for each section is as follows: to present the historical context overview; to examine the significance of IoT and its application in smart cities; to present a standardized architecture for IoT-based smart city surveillance; to highlight an authoritative and thorough review of current IoT-based smart surveillance systems; and to identify various research issues. The methods used are: statistical graph or chart approach, schematic, and timeline diagram. Conclusions. This article outlines numerous research challenges for future video surveillance that may be addressed by researchers. In summary, this comprehensive review provides a valuable and streamlined resource for future researchers exploring smart city surveillance through the IoT.

Continuous Authentication for Consumer Electronics in Smart City Surveillance

Continuous Authentication for Consumer Electronics in Smart City Surveillance

AI-based outdoor moving object detection for smart city surveillance

<abstract> <p>One essential component of the futuristic way of living in "smart cities" is the installation of surveillance cameras. There are a wide variety of applications for surveillance cameras, including but not limited to: investigating and preventing crimes, identifying sick individuals (coronavirus), locating missing persons, and many more. In this research, we provided a system for smart city outdoor item recognition using visual data collected by security cameras. The object identification model used by the proposed outdoor system was an enhanced version of RetinaNet. A state of the art object identification model, RetinaNet boasts lightning-fast processing and pinpoint accuracy. Its primary purpose was to rectify the focal loss-based training dataset's inherent class imbalance. To make the RetinaNet better at identifying tiny objects, we increased its receptive field with custom-made convolution blocks. In addition, we adjusted the number of anchors by decreasing their scale and increasing their ratio. Using a mix of open-source datasets including BDD100K, MS COCO, and Pascal Vocab, the suggested outdoor object identification system was trained and tested. While maintaining real-time operation, the suggested system's performance has been markedly enhanced in terms of accuracy.</p> </abstract>

Crowd Emotion Prediction for Human-Vehicle Interaction Through Modified Transfer Learning and Fuzzy Logic Ranking

In metropolitan environments, unmanned aerial vehicles (UAVs) equipped with video surveillance equipment can monitor crowd behavior and maintain public safety. In high-traffic areas where humans are more likely to make mistakes, a smart city needs modern technology to forecast the behavior of its residents. In order to improve citywide traffic flow, urban transportation systems (UTSs) monitor and learn how people behave in crowds. Using UAVs for video surveillance in smart cities, our research describes a unique way to assess crowd condition, which expands the scope of human-vehicle interactions. Moreover, we use fuzzy logic ranking to improve the system’s ability to detect anomalies in crowds. In order to improve decision-making, a novel deep transfer learning (DTL) technique is applied to the UAV’s received frames. A 98.5% accuracy rate, satisfactory performance, and robustness to population behavior are all characteristics of the proposed integrated model. In UTSs and urban areas, our novel intelligent system analyzes human behavior based on vehicle-human interactions. In areas with low and high traffic congestion, the modified ResNet (mResNet) architecture predicts the crowd’s condition based on fuzzy logic (FLA). Through decision-making based on accurate crowd conditions, best general paths can be selected using the algorithm.

A Novel Unsupervised Video Anomaly Detection Framework Based on Optical Flow Reconstruction and Erased Frame Prediction.

Reconstruction-based and prediction-based approaches are widely used for video anomaly detection (VAD) in smart city surveillance applications. However, neither of these approaches can effectively utilize the rich contextual information that exists in videos, which makes it difficult to accurately perceive anomalous activities. In this paper, we exploit the idea of a training model based on the "Cloze Test" strategy in natural language processing (NLP) and introduce a novel unsupervised learning framework to encode both motion and appearance information at an object level. Specifically, to store the normal modes of video activity reconstructions, we first design an optical stream memory network with skip connections. Secondly, we build a space-time cube (STC) for use as the basic processing unit of the model and erase a patch in the STC to form the frame to be reconstructed. This enables a so-called "incomplete event (IE)" to be completed. On this basis, a conditional autoencoder is utilized to capture the high correspondence between optical flow and STC. The model predicts erased patches in IEs based on the context of the front and back frames. Finally, we employ a generating adversarial network (GAN)-based training method to improve the performance of VAD. By distinguishing the predicted erased optical flow and erased video frame, the anomaly detection results are shown to be more reliable with our proposed method which can help reconstruct the original video in IE. Comparative experiments conducted on the benchmark UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets demonstrate AUROC scores reaching 97.7%, 89.7%, and 75.8%, respectively.

Citizen empowerment through smart surveillance: evidence from Indian smart cities

PurposeCitizens often perceive surveillance by government authorities as oppressive and, hence, demonstrate reluctance in value co-creation from such services. This study aims to investigate the challenges and benefits of citizen empowerment through technology-driven surveillance or “smart surveillance.”Design/methodology/approachGuided by Dynamic Capability theory, the authors conduct in-depth interviews with officers in-charge of surveillance in smart cities. Given the contemporary advancements, this approach allows a retrospective and real-time understanding of interviewees’ experiences with smart surveillance.FindingsThe authors develop five propositions for citizen empowerment through smart surveillance to summarize the findings of this study.Research limitations/implicationsThis study advances the relevance of Dynamic Capability in public administration.Practical implicationsSmart city authorities and policymakers may leverage the insights provided in this study to design appropriate policies for smart surveillance.Originality/valueThe authors find that factors such as digital technology and infrastructure, information management, skill divide and perceived return on investment may influence citizen empowerment through smart surveillance.

Making Smart Things Strange Again: Using Walking as a Method for Studying Subjective Experiences of Smart City Surveillance

Smart cities are commonly seen as places that are defined by surveillance because of their reliance on vast amounts of digital data to improve urban management challenges. Although the infrastructures and technologies that enable smart city surveillance pervade multitudinous urban spaces and everyday practices, they are often “hiding in plain sight,” going unnoticed in the bustle of everyday life. Hence, fostering research settings where citizens can productively reflect on their everyday surveillance constitutes a major challenge for the interrelated projects of doing empirical research about subjective experiences of smart city surveillance and the inclusion of citizens in smart city discussions. Drawing on walking as a method, this study attempts to meet this challenge by developing and empirically testing a methodology of purposive “data walking.” Situating the research in Rotterdam, the Netherlands, participants are instructed to identify data points for public safety purposes on a short walk through the city and reflect on their experiences. Observations and experiences of smart city surveillance are documented with photos, text descriptions, and audio notes, which are shared in real-time with researchers and provide the basis for group reflections. These walks and reflections generate rich visual and textual data that yield insights into embodied and situated constructions of smart city surveillance as an object of subjective inquiry, experiences of visibility, considerations of agency and evaluations of public safety implications. The study considers these empirical results in conjunction with reflections on the methodology, contributing to further methodological explorations for including citizens in smart city discussions and surveillance subjectivity research.

Vision transformer attention with multi-reservoir echo state network for anomaly recognition

Anomalous event recognition requires an instant response to reduce the loss of human life and property; however, existing automated systems show limited performance due to considerations related to the temporal domain of the videos and ignore the significant role of spatial information. Furthermore, although current surveillance systems can detect anomalous events, they require human intervention to recognise their nature and to select appropriate countermeasures, as there are no fully automatic surveillance techniques that can simultaneously detect and interpret anomalous events. Therefore, we present a framework called Vision Transformer Anomaly Recognition (ViT-ARN) that can detect and interpret anomalies in smart city surveillance videos. The framework consists of two stages: the first involves online anomaly detection, for which a customised, lightweight, one-class deep neural network is developed to detect anomalies in a surveillance environment, while in the second stage, the detected anomaly is further classified into the corresponding class. The size of our anomaly detection model is compressed using a filter pruning strategy based on a geometric median, with the aim of easy adaptability for resource-constrained devices. Anomaly classification is based on vision transformer features and is followed by a bottleneck attention mechanism to enhance the representation. The refined features are passed to a multi-reservoir echo state network for a detailed analysis of real-world anomalies such as vandalism and road accidents. A total of 858 and 1600 videos from two datasets are used to train the proposed model, and extensive experiments on the LAD-2000 and UCF-Crime datasets comprising 290 and 400 testing videos reveal that our framework can recognise anomalies more effectively, outperforming other state-of-the-art approaches with increases in accuracy of 10.14% and 3% on the LAD-2000 and UCF-Crime datasets, respectively.

An efficient deep learning-assisted person re-identification solution for intelligent video surveillance in smart cities

An efficient deep learning-assisted person re-identification solution for intelligent video surveillance in smart cities

Comments on “A Secure and Lightweight Drones-Access Protocol for Smart City Surveillance”

Recently, Akram et al. (2021) proposed a drones-access protocol for monitoring urban security. Akram et al. claimed that their protocol is secure and can resist known security attacks. However, in this comment article, we demonstrate that their protocol is still vulnerable to drone capture attacks and stolen-verifier attacks. In addition, their protocol does not provide perfect forward secrecy. Finally, we present an improved protocol to manage the security weaknesses we found.

Aquila Optimization with Transfer Learning Based Crowd Density Analysis for Sustainable Smart Cities

Video surveillance in smart cities provides efficient city operations, safer communities, and improved municipal services. Object detection is a computer vision-based technology, which is utilized for detecting instances of semantic objects of a specific class in digital videos and images. Crowd density analysis is a widely used application of object detection, while crowd density classification techniques face complications such as inter-scene deviations, non-uniform density, intra-scene deviations and occlusion. The convolution neural network (CNN) model is advantageous. This study presents Aquila Optimization with Transfer Learning based Crowd Density Analysis for Sustainable Smart Cities (AOTL-CDA3S). The presented AOTL-CDA3S technique aims to identify different kinds of crowd densities in the smart cities. For accomplishing this, the proposed AOTL-CDA3S model initially applies a weighted average filter (WAF) technique for improving the quality of the input frames. Next, the AOTL-CDA3S technique employs an AO algorithm with the SqueezeNet model for feature extraction. Finally, to classify crowd densities, an extreme gradient boosting (XGBoost) classification model is used. The experimental validation of the AOTL-CDA3S approach is tested by means of benchmark crowd datasets and the results are examined under distinct metrics. This study reports the improvements of the AOTL-CDA3S model over recent state of the art methods.

Internet-of-Things-Based Suspicious Activity Recognition Using Multimodalities of Computer Vision for Smart City Security

Automatic human activity recognition is one of the milestones of smart city surveillance projects. Human activity detection and recognition aim to identify the activities based on the observations that are being performed by the subject. Hence, vision-based human activity recognition systems have a wide scope in video surveillance, health care systems, and human-computer interaction. Currently, the world is moving towards a smart and safe city concept. Automatic human activity recognition is the major challenge of smart city surveillance. The proposed research work employed fine-tuned YOLO-v4 for activity detection, whereas for classification purposes, 3D-CNN has been implemented. Besides the classification, the presented research model also leverages human-object interaction with the help of intersection over union (IOU). An Internet of Things (IoT) based architecture is implemented to take efficient and real-time decisions. The dataset of exploit classes has been taken from the UCF-Crime dataset for activity recognition. At the same time, the dataset extracted from MS-COCO for suspicious object detection is involved in human-object interaction. This research is also applied to human activity detection and recognition in the university premises for real-time suspicious activity detection and automatic alerts. The experiments have exhibited that the proposed multimodal approach achieves remarkable activity detection and recognition accuracy.

Internet of Things-Enabled Optimal Data Aggregation Approach for the Intelligent Surveillance Systems

The Internet of Things (IoT)-based intelligent surveillance systems in smart cities are a challenging issue as various devices capture the data. These devices, deployed close to the underlined phenomenon, such as cameras, are duplicated or redundant as accuracy is the main requirement of these systems. For this purpose, sensor nodes are deployed to provide 24/7 monitoring of a smart city, which minimizes the security risks and enables quick response in case of any disaster. However, due to a large number of devices, huge data are generated; thus, controlling traffic congestion in case of undesirable circumstances, for example, in case of accident or intention blockage of the road, is desperately needed. Numerous data aggregation mechanisms were reported in the literature to address this issue with smart city surveillance systems. However, these approaches were designed for either specific application environments or complex environments, making the implementation process hard. In this article, we have developed an Internet of Things-enabled optimal data aggregation approach, specifically designed for the intelligent surveillance systems in smart cities, to convert raw data values into the refined ones with minimum possible data loss ratio. Moreover, the proposed scheme bounds every server to perform or carry out the data refinement process to maintain the expected ratio of accuracy and precision. In this approach, ordinary devices are forced to capture and forward data in raw form, preferably without any or minimum possible processing. This reduces the load on ordinary devices in intelligent surveillance systems. Additionally, we have developed a novel approach to eliminate or reduce (if elimination is not possible) noisy data or outliers. It is implemented along with existing state-of-the-art techniques to verify the exceptional performance of the proposed data aggregation approach. These algorithms were compared using various performance evaluation metrics such as refinement ratio, data loss ratio, energy efficiency, and lifetime. The simulation results have verified that the proposed scheme’s performance is better than the existing approaches.

IoT intelligence empowered by end-edge-cloud orchestration

To support intelligent Internet of Things (IoT) applications, such as autonomous driving, smart city surveillance, and virtual reality (VR)/augmented reality (AR), cloud services are expected to be pushed to the proximity of IoT devices for quality performance. For instance, to facilitate safe autonomous driving, the service delay of most vehicular applications is required to be within milliseconds, and any information delay may result in dangerous on-road conditions. Edge intelligence aims at processing data/computing-intensive IoT tasks at the edge of network, where a set of IoT devices can work cooperatively for data collection, processing, model training, caching, and data analytics via edge caching, edge training, edge offloading, etc. It empowers intelligent IoT services at the network edge. However, it is challenging to achieve satisfying performance due to the following reasons. On the one hand, as the edge nodes are constrained by storage/computing resources, it is essential to conduct end-edge-cloud resource orchestration and resource sharing, taking into account device mobility, burst & stochastic service requests, and heterogeneous resources. On the other hand, better quality of service/learning of IoT applications is difficult to be guaranteed as the task execution/model training is contributed by distributed IoT devices. As a result, individual quality characterization, participating node selection, multi-level collaboration, robustness against malicious attacks are crucial. At last, to enable IoT intelligence, frequent communication and coordination among IoT devices, edge nodes and cloud servers are required, which can raise significant overhead, delay, and potential disclosure of sensitive information. Overcoming those challenges calls for further in-depth research.

Video Streaming Adaptive QoS Routing with Resource Reservation (VQoSRR) Model for SDN Networks

Video streaming has become extremely widespread, especially with the growing number of users and the spread of mobile devices, along with the increase in the availability and diversity of multimedia applications and communication technologies. Real-time video communication requires awareness of the quality of experience (QoE) to provide customers with a satisfactory service, for example, in smart cities that use video surveillance systems. The quality of service (QoS) is dependent on network performance, which directly affects the QoE. However, reliance on traditional network infrastructure and routing protocols cannot assure QoS. The emergence of software defined networks (SDN) may eliminate current network limitations. Due to SDN’s global view and programmability characteristics, such capabilities could help in providing an automated QoS control and management. This paper introduces video streaming adaptive QoS-based routing and resource reservation (VQoSRR), which gives SDN networks the ability to meet video demands and enhance user experience over best effort networks, such as the ones required for video surveillance in smart cities. In order to implement QoS-based routing (QBR), we developed algorithms for calculating routing, installing routing paths in the forwarding devices, and shifting traffic to an alternative path when QoE is violated. As well, we used queuing mechanisms to allocate resources based on the QoE requirements of video streaming. Our results indicate that resource reservation mechanisms combined with QoS-based routing enable effective control over routes and resources. Our framework guarantees the video quality as well. This technique of using video streaming would improve the tools and applications used for smart cities such as surveillance systems for hospitals and civil defense organizations.

Distributed Real-Time Object Detection Based on Edge-Cloud Collaboration for Smart Video Surveillance Applications

Internet of Things (IoT) and artificial intelligence (AI) can realize the concept of “smart city.” Video surveillance in smart cities is, usually, based on a centralized framework in which large amounts of real-time media data are transmitted to and processed in the cloud. However, the cloud relies on network connectivity of the Internet that is sometimes limited or unavailable; thus, the centralized framework is not sufficient for real-time processing of media data needed for smart video surveillance. To tackle this problem, edge computing - a technique for accelerating the development of AIoT (AI across IoT) in smart cities - can be conducted. In this paper, a distributed real-time object detection framework based on edge-cloud collaboration for smart video surveillance is proposed. When collaborating with the cloud, edge computing can serve as converged computing through which media data from distributed edge devices of the network are consolidated by AI in the cloud. After AI discovers global knowledge in the cloud, it to be shared at the edge is deployed remotely on distributed edge devices for real-time smart video surveillance. First, the proposed framework and its preliminary implementation are described. Then, the performance evaluation is provided regarding potential benefits, real-time responsiveness and low-throughput media data transmission.