Real-time Constraints Research Articles

Real-Time Smoke Detection in Surveillance Videos Using an Enhanced RT-DETR Framework with Triplet Attention and HS-FPN

This study addresses the urgent need for an efficient and accurate smoke detection system to enhance safety measures in fire monitoring, industrial safety, and urban surveillance. Given the complexity of detecting smoke in diverse environments and under real-time constraints, our research aims to solve challenges related to low-resolution imagery, limited computational resources, and environmental variability. This study introduces a novel smoke detection system that utilizes the real-time detection Transformer (RT-DETR) architecture to enhance the speed and precision of video analysis. Our system integrates advanced modules, including triplet attention, ADown, and a high-level screening-feature fusion pyramid network (HS-FPN), to address challenges related to low-resolution imagery, real-time processing constraints, and environmental variability. The triplet attention mechanism is essential for detecting subtle smoke features, often overlooked due to their nuanced nature. The ADown module significantly reduces computational complexity, enabling real-time operation on devices with limited resources. Furthermore, the HS-FPN enhances the system’s robustness by amalgamating multi-scale features for reliable detection across various smoke types and sizes. Evaluation using a diverse dataset showcased notable improvements in average precision (AP50) and frames per second (FPS) metrics compared to existing state-of-the-art networks. Ablation studies validated the contributions of each component in achieving an optimal balance between accuracy and operational efficiency. The RT-DETR-based smoke detection system not only meets real-time requirements for applications like fire monitoring, industrial safety, and urban surveillance but also establishes a new performance benchmark in this field.

Optimization of Truck–Cargo Matching for the LTL Logistics Hub Based on Three-Dimensional Pallet Loading

This study investigates the truck–cargo matching problem in less-than-truckload (LTL) logistics hubs, focusing on optimizing the three-dimensional loading of goods onto standardized pallets and assigning these loaded pallets to a fleet of heterogeneous vehicles. A two-stage hybrid heuristic algorithm is proposed to solve this complex logistics challenge. In the first stage, a tree search algorithm based on residual space is developed to determine the optimal layout for the 3D loading of cargo onto pallets. In the second stage, a heuristic online truck–cargo matching algorithm is introduced to allocate loaded pallets to trucks while optimizing the number of trucks used and minimizing transportation costs. The algorithm operates within a rolling time horizon, allowing it to dynamically handle real-time order arrivals and time window constraints. Numerical experiments demonstrate that the proposed method achieves high pallet loading efficiency (close to 90%), near-optimal truck utilization (nearly 95%), and significant cost reductions, making it a practical solution for real-world LTL logistics operations.

Joint range and velocity super-resolution estimation with Doppler effects for innovative OFDM-based RFA RadCom system

Conventional radar and communication signals face challenges when integrating for accurate sensing in dense electromagnetic environments, especially in scenarios involving high-velocity targets estimation. To address this issue, we propose the random frequency agile-orthogonal frequency division multiplexing-based radar and communication (RFA-OFDM-based RadCom) signal, a novel framework that combines RFA hopping radar signal and OFDM signal. This framework effectively handles high-velocity Doppler scenarios, enhancing electronic countermeasure capabilities. In high-velocity scenarios, achieving accurate range and velocity estimation is crucial. We introduce a comprehensive received signal model that considers intrapulse and intersubcarrier Doppler effects, often overlooked in traditional high-velocity contexts. The proposed two-phase hierarchical perceptual methodology enables joint super-resolution estimation using the shared signal. We transform the shared signal echo model into a uniform linear array-like model and employ the matrix decomposition algorithm based on bidirectional weighted frequency smoothing (BWFS-MD) for decoherence processing. Subsequently, the estimation of signal parameters via rotational invariance techniques (ESPRIT)-complementary integrated subspace fitting (E-CISF) algorithm accurately estimates joint range and velocity. Meanwhile, the contrastive analysis of the mutual impacts between radar and communication functions is conducted. Theoretical analysis and simulation results robustly validate the superior performance of the proposed BWFS-MD algorithm. Furthermore, considering the precision of joint range-velocity estimation, real-time constraints, and super-resolution capability (which is emphasized), the E-CSIF algorithm demonstrates the best overall performance from a comprehensive perspective.

Lateral-Stability-Oriented Path-Tracking Control Design for Four-Wheel Independent Drive Autonomous Vehicles with Tire Dynamic Characteristics under Extreme Conditions

This paper proposes a lateral-stability-oriented path-tracking controller for four-wheel independent drive (4WID) autonomous vehicles. The proposed controller aims to maintain vehicle stability under extreme conditions while minimizing lateral deviation. Firstly, a tiered control framework comprising upper-level and lower-level controllers is introduced. The upper-level controller is a lateral stability path-tracking controller that incorporates tire dynamic characteristics, developed using model predictive control (MPC) theory. This controller dynamically updates the tire lateral force constraints in real time to account for variations in tire dynamics under extreme conditions. Additionally, it enhances lateral stability and reduces path-tracking errors by applying additional yaw torque based on minimum tire utilization. The lower-level controllers execute the required steering angles and yaw moments through the appropriate component equipment and torque distribution. The joint simulation results from CarSim and MATLAB/Simulink show that, compared to the traditional MPC controller with unstable sideslip, this controller can maintain vehicle lateral stability under extreme conditions. Compared to the MPC controller, which only considers lateral force constraints, this controller can significantly reduce lateral tracking errors, with an average yaw rate reduction of 31.62% and an average sideslip angle reduction of 40.21%.

Mobility-Assisted Digital Twin Network Optimization over Industrial Internet of Things

Many real-world networks for the Industrial Internet of Things (IIoT) have diverse connectivity characteristics and real-time constraints imposed by industrial processing. In the context of digital twin networks (DTNs), a large number of IIoT devices may access the network and have a tremendous volume of data. A crucial element of these IIoT devices is mobility, which cannot be effectively solved because the number of IIoT devices connected is extremely large. IIoT devices in DTNs suffer from poor data transmission and link quality because of their mobility. In this paper, device-to-device (D2D) communication-based mobility-assisted digital twin networks are proposed, where edge computing is introduced to design an efficient mapping between the physical and virtual space. Then, we propose the architecture of data transmission for the D2D network to maximize the data rate for reliable connectivity among multiple mobile nodes based on IIoT. A Markov decision process (MDP) is formulated to maximize the data rate for multiple mobile nodes while maintaining the D2D communication link quality. The simulation results demonstrate the superiority of the proposed scheme over other comparable models.

A Novel Approach for Trajectory Tracking Control of an Under-Actuated Quad-Rotor UAV

A novel Udwadia-Kalaba approach for the trajectory tracking control of an under-actuated quad-rotor unmanned aerial vehicle (UAV) is presented. Compared to standard control approaches, the desired trajectories are treated as constraints called trajectory tracking constraints in this approach. Neither making any approximations or linearization of the nonlinear system nor imposing any a priori structure on the nature of the nonlinear controller, this methodology provides closedform nonlinear control. The control inputs satisfying the desired trajectory requirements can be obtained explicitly in compact closed form by solving Udwadia-Kalaba equation. Nonlinear dynamics modeling of a quad-rotor UAV is processed and a desired trajectory is given in this paper to illustrate this approach. The theoretical analysis and MATLAB simulation results verify the validity and efficiency of this approach. The real-time servo constraint forces are obtained conveniently and the quad-rotor UAV's movement meets the designed trajectory precisely.

Interactive CAD Layout of Reflection-Based Mirror Metrology Systems

Concentrating solar mirrors need to accurately focus sunlight onto a receiver. Since mirror surface normal determines reflected light direction, reflection-based metrology methods that measure surface normal have the advantage of directly measuring the mirror’s intended functional performance characteristic. Several methods have been reported that produce high-resolution measurements of mirror surface normal of concentrating solar mirrors [1-8]; these systems typically employ an optical target, a camera, and sometimes an external light source. These elements must be placed in a configuration that satisfies multiple feasibility constraints, including mechanical and optical visibility and reflection constraints. For complex problems or installations in cluttered environments, it can be difficult to analyze all of these constraints manually. We have created a computer automated design (CAD) tool that aids in visualization of reflection-based metrology systems, making it possible to determine layouts for complex metrology design problems. The tool is implemented SolidWorks, a commercial CAD system [9]. By exploiting the SoidWorks constraint satisfaction mechanism, the tool is able to present interactive visualization of reflection and field of view constraints in real time without any additional code or macros. The tool supports ease of use and multiple design configurations through an ordinary spreadsheet interface. The tool can be used to address a variety of reflection-based metrology design problems, and is available to the public as open source.

PyIgH : A unified architecture of IgH EtherCAT Master based on Python considering hard real-time constraints

The increasing demand for rapid application development tools, especially those employing high-level languages such as Python, has underscored the importance of utilizing a wide array of popular libraries while addressing real-time constraints in distributed hardware systems. This paper introduces PyIgH, a unified architecture of an IgH EtherCAT master based on Python, specifically designed to satisfy hard real-time requirements in an EtherCAT network. Implemented as a Python module, PyIgH exposes the functionalities and capabilities of an open-source EtherCAT master, facilitating seamless configuration and control of EtherCAT slave devices within the Python runtime environment. Real-time adaptation of the POSIX library, encapsulated within Python, is also utilized to satisfy the timing requirements of EtherCAT. The feasibility of the proposed approach is verified by analyzing the real-time performance in terms of periodicity and in-controller delay of the EtherCAT control task with a 1 kHz cycle. Experimental results demonstrate that PyIgH is suitable for hard real-time applications and serves as a valid alternative to conventional low-level EtherCAT masters. Additionally, a practical application involving motion control of a six-axis collaborative robot showcases consistent performance of PyIgH within a real-time multi-tasking environment.

Efficient Management of Containers for Software Defined Vehicles

Containerization technology, such as Docker, is gaining in popularity in newly established software-defined vehicle architectures (SDVA). However, executing those containers can quickly become computationally expensive in constrained environments, given the limited CPU, memory, and energy resources in the Electric Control Units (ECU) of SDVA. Consequently, the efficient management of these containers is crucial for enabling the on-demand usage of the applications in the vehicle based on the available resources while considering several constraints and priorities, including failure tolerance, security, safety, and comfort. In this paper, we propose a dynamic software container management approach for constrained environments such as embedded devices/ECUs in SDVA within smart cars. To address the conflicting objectives and constraints within the vehicle, we design a novel search-based approach based on multi-objective optimization. This approach facilitates the allocation, movement, or suspension of containers between ECUs in the cluster. Collaborating with our industry partner, Ford Motor Company, we evaluate our approach using different real-world software-defined scenarios. These scenarios involve using heterogeneous clusters of ECU devices in vehicles based on real-world software containers and use-case studies from the automotive industry. The experimental results demonstrate that our scheduler outperforms existing scheduling algorithms, including the default Docker scheduler -Spread- commonly used in automotive applications. Our proposed scheduler exhibits superior performance in terms of energy and resource cost efficiency. Specifically, it achieves a 35% reduction in energy consumption in power-saving mode compared to the scheduler employed by Ford Motor Company. Additionally, our scheduler effectively distributes workload among the ECUs in the cluster, minimizing resource usage, and dynamically adjusts to the real-time requirements and constraints of the car environment. This work will serve as a fundamental building block in the automotive industry to efficiently manage software containers in smart vehicles considering constraints and priorities in the real world.

LExCI: A framework for reinforcement learning with embedded systems

Advances in artificial intelligence (AI) have led to its application in many areas of everyday life. In the context of control engineering, reinforcement learning (RL) represents a particularly promising approach as it is centred around the idea of allowing an agent to freely interact with its environment to find an optimal strategy. One of the challenges professionals face when training and deploying RL agents is that the latter often have to run on dedicated embedded devices. This could be to integrate them into an existing toolchain or to satisfy certain performance criteria like real-time constraints. Conventional RL libraries, however, cannot be easily utilised in conjunction with that kind of hardware. In this paper, we present a framework named LExCI, the Learning and Experiencing Cycle Interface, which bridges this gap and provides end-users with a free and open-source tool for training agents on embedded systems using the open-source library RLlib. Its operability is demonstrated with two state-of-the-art RL-algorithms and a rapid control prototyping system.

A Real-Time Resource Dispatch Approach for Edge Computing Devices in Digital Distribution Networks Considering Burst Tasks

Edge computing technology can effectively solve huge challenges posed by the large number of terminal devices accessing and massive data processing in digital distribution networks. Burst tasks, such as faults and data requests from the cloud, can occur at any time for edge computing devices in distribution networks. These tasks are unpredictable and usually hold the highest priority and must be completed as soon as possible. Although resources can be reserved partially at each period in the pre-scheduled operation plan, they may still be insufficient to handle burst tasks adequately. A real-time resource dispatch approach for burst tasks is developed in this study to address the above problems. The concept of flexibility for edge computing devices is presented, determining the real-time dispatch duration. Real-time resource dispatch and task handling processing are analyzed in detail, considered as task real-time dispatch models, computation process real-time dispatch constraints, and resource limitation constraints. The proposed real-time resource dispatch approach takes full advantage of the transferable characteristics for partial original plan tasks to adjust the pre-scheduled operation plan and release flexible resources for immediate processing of the burst task, completing burst tasks quickly and minimizing the impact for previous planned tasks on the edge computing device. The capability of the proposed method to efficiently deal with the burst tasks is also verified by the case study.

A lightweight fire detection algorithm for small targets based on YOLOv5s

In response to the current challenges fire detection algorithms encounter, including low detection accuracy and limited recognition rates for small fire targets in complex environments, we present a lightweight fire detection algorithm based on an improved YOLOv5s. The introduction of the CoT (Contextual Transformer) structure into the backbone neural network, along with the creation of the novel CSP1_CoT (Cross stage partial 1_contextual transformer) module, has effectively reduced the model’s parameter count while simultaneously enhancing the feature extraction and fusion capabilities of the backbone network; The network’s Neck architecture has been extended by introducing a dedicated detection layer tailored for small targets and incorporating the SE (Squeeze-and-Excitation) attention mechanism. This augmentation, while minimizing parameter proliferation, has significantly bolstered the interaction of multi-feature information, resulting in an enhanced small target detection capability; The substitution of the original loss function with the Focal-EIoU (Focal-Efficient IoU) loss function has yielded a further improvement in the model’s convergence speed and precision; The experimental results indicate that the modified model achieves an mAP@.5 of 96% and an accuracy of 94.8%, marking improvements of 8.8% and 8.9%, respectively, over the original model. Furthermore, the model’s parameter count has been reduced by 1.1%, resulting in a compact model size of only 14.6MB. Additionally, the detection speed has reached 85 FPS (Frames Per Second), thus satisfying real-time detection requirements. This enhancement in precision and accuracy, while simultaneously meeting real-time and lightweight constraints, effectively caters to the demands of fire detection.

Real-Time Trajectory Smoothing and Obstacle Avoidance: A Method Based on Virtual Force Guidance.

In dynamic environments, real-time trajectory planners are required to generate smooth trajectories. However, trajectory planners based on real-time sampling often produce jerky trajectories that necessitate post-processing steps for smoothing. Existing local smoothing methods may result in trajectories that collide with obstacles due to the lack of a direct connection between the smoothing process and trajectory optimization. To address this limitation, this paper proposes a novel trajectory-smoothing method that considers obstacle constraints in real time. By introducing virtual attractive forces from original trajectory points and virtual repulsive forces from obstacles, the resultant force guides the generation of smooth trajectories. This approach enables parallel execution with the trajectory-planning process and requires low computational overhead. Experimental validation in different scenarios demonstrates that the proposed method not only achieves real-time trajectory smoothing but also effectively avoids obstacles.

A Virtual Testing Framework for Real-Time Validation of Automotive Software Systems Based on Hardware in the Loop and Fault Injection.

To validate safety-related automotive software systems, experimental tests are conducted at different stages of the V-model, which are referred as "X-in-the-loop (XIL) methods". However, these methods have significant drawbacks in terms of cost, time, effort and effectiveness. In this study, based on hardware-in-the-loop (HIL) simulation and real-time fault injection (FI), a novel testing framework has been developed to validate system performance under critical abnormal situations during the development process. The developed framework provides an approach for the real-time analysis of system behavior under single and simultaneous sensor/actuator-related faults during virtual test drives without modeling effort for fault mode simulations. Unlike traditional methods, the faults are injected programmatically and the system architecture is ensured without modification to meet the real-time constraints. Moreover, a virtual environment is modeled with various environmental conditions, such as weather, traffic and roads. The validation results demonstrate the effectiveness of the proposed framework in a variety of driving scenarios. The evaluation results demonstrate that the system behavior via HIL simulation has a high accuracy compared to the non-real-time simulation method with an average relative error of 2.52. The comparative study with the state-of-the-art methods indicates that the proposed approach exhibits superior accuracy and capability. This, in turn, provides a safe, reliable and realistic environment for the real-time validation of complex automotive systems at a low cost, with minimal time and effort.

On [formula omitted]-performance of weakly-hard real-time control systems

This paper considers control systems with failures in the feedback channel, that occasionally lead to loss of the control input signal. A useful approach for modeling such failures is to consider window-based constraints on possible loss sequences, for example that at least r control attempts in every window of s are successful. A powerful framework to model such constraints are weakly-hard real-time constraints. Various approaches for stability analysis and the synthesis of stabilizing controllers for such systems have been presented in the past. However, existing results are mostly limited to asymptotic stability and rarely consider performance measures such as the resulting ℓ2-gain. To address this problem, we adapt a switched system description where the switching sequence is constrained by a graph that captures the loss information. We present an approach for ℓ2-performance analysis involving linear matrix inequalities (LMI). Further, leveraging a system lifting method, we propose an LMI-based approach for synthesizing state-feedback controllers with guaranteed ℓ2-performance. The results are illustrated by a numerical example.

Robust digital-twin airspace discretization and trajectory optimization for autonomous unmanned aerial vehicles

The infiltration of heterogenous fleets of autonomous Unmanned Aerial Vehicles (UAVs) in smart cities is leading to the consumerization of city air space which includes infrastructure creation of roads, traffic design, capacity estimation, and trajectory optimization. This study proposes a novel autonomous Advanced Aerial Mobility (AAM) logistical system for high density city centers. First, we propose a real-time 3D geospatial mining framework for LiDAR data to create a dynamically updated digital twin model. This enables the identification of viable airspace volumes in densely populated 3D environments based on the airspace policy/regulations. Second, we propose a robust city airspace dynamic 4D discretization method (Skyroutes) for autonomous UAVs to incorporate the underlying real-time constraints coupled with externalities, legal, and optimal UAV operation based on kinematics. An hourly trip generation model was applied to create 1138 trips in two scenarios comparing the cartesian discretization to our proposed algorithm. The results show that the AAM enables a precise airspace capacity/cost estimation, due to its detailed 3D generation capabilities. The AAM increased the airspace capacity by up to 10%, the generated UAV trajectories are 50% more energy efficient, and significantly safer.

A machine learning estimator trained on synthetic data for real-time earthquake ground-shaking predictions in Southern California

After large-magnitude earthquakes, a crucial task for impact assessment is to rapidly and accurately estimate the ground shaking in the affected region. To satisfy real-time constraints, intensity measures are traditionally evaluated with empirical Ground Motion Models that can drastically limit the accuracy of the estimated values. As an alternative, here we present Machine Learning strategies trained on physics-based simulations that require similar evaluation times. We trained and validated the proposed Machine Learning-based Estimator for ground shaking maps with one of the largest existing datasets (<100M simulated seismograms) from CyberShake developed by the Southern California Earthquake Center covering the Los Angeles basin. For a well-tailored synthetic database, our predictions outperform empirical Ground Motion Models provided that the events considered are compatible with the training data. Using the proposed strategy we show significant error reductions not only for synthetic, but also for five real historical earthquakes, relative to empirical Ground Motion Models.

Design and Development of an Intuitive Android Application for Smart Farming

Farmers face challenges in adapting to environmental changes, yet with decreasing implementation costs and a growing accessibility to technical expertise, progress is evident. This research introduces KisanVishwa, an Android app designed for smart farming. KisanVishwa optimizes precision agriculture systems, enabling farmers to monitor their fields in real time and address agricultural constraints. Tailored to meet the specific needs of today's farmers, the app provides user-friendly tools and the latest information on farm equipment, fertilizers, chemicals, technological advancements, and local distributors and retailers. By integrating KisanVishwa into their operations, farmers can overcome challenges associated with the adoption and maintenance of precision agriculture. KisanVishwa adopts a user-centric design for compatibility across diverse Android devices, integrating monitoring tools and collaborating with agricultural research institutions. Empowering farmers, the app addresses contemporary challenges, fostering informed decisions and efficiency. With its user-friendly interface, KisanVishwa promotes the adoption of precision agriculture, essential resources, and sustainable practices in the agricultural sector. Convincing traditional farmers to adopt advanced technologies may pose challenges. Functionality may depend on reliable internet, posing challenges in areas with limited connectivity.

Real-time motion estimation based video steganography with preserved consistency and local optimality

AbstractThis paper introduces a novel steganographic technique for H.264 video that achieves an outstanding performance against state-of-the-art steganalysis techniques while maintaining real-time encoding performance constraints. The proposed technique embeds the secret message by altering the motion vectors (MVs) while preserving their local optimality and consistency feature to withstand the recently emerged steganalysis methods. Thanks to its macro-block (MB) basis architecture, the proposed technique satisfies the real-time constraints, eliminating the need to wait for the whole frame or group of pictures (GOP) and avoiding the need to perform any additional re-encoding step(s). Additionally, altering the MVs is performed in the motion estimation (ME) sub-pixel-refinement stage through a rule-based scheme that ensures each MB’s compatibility for embedding without detection by the aforementioned steganalysis methods. The proposed technique is integrated with the OpenH264 real-time video encoder and evaluated on widely used video sequences. The results prove that the proposed technique achieves a significant security performance against the steganalysis methods while maintaining an acceptable embedding rate, outperforming other state-of-the-art MV-based steganographic methods in real-time constrained environments. The proposed technique adds about $$1-2\%$$ 1 - 2 % overhead beyond the encoder running time. The source code is publicly available here: https://github.com/HassanMohamedGit/OpenH264-RealTime-steg.

Real-time active constraint generation and enforcement for surgical tools using 3D detection and localisation network.

Introduction: Collaborative robots, designed to work alongside humans for manipulating end-effectors, greatly benefit from the implementation of active constraints. This process comprises the definition of a boundary, followed by the enforcement of some control algorithm when the robot tooltip interacts with the generated boundary. Contact with the constraint boundary is communicated to the human operator through various potential forms of feedback. In fields like surgical robotics, where patient safety is paramount, implementing active constraints can prevent the robot from interacting with portions of the patient anatomy that shouldn't be operated on. Despite improvements in orthopaedic surgical robots, however, there exists a gap between bulky systems with haptic feedback capabilities and miniaturised systems that only allow for boundary control, where interaction with the active constraint boundary interrupts robot functions. Generally, active constraint generation relies on optical tracking systems and preoperative imaging techniques. Methods: This paper presents a refined version of the Signature Robot, a three degrees-of-freedom, hands-on collaborative system for orthopaedic surgery. Additionally, it presents a method for generating and enforcing active constraints "on-the-fly" using our previously introduced monocular, RGB, camera-based network, SimPS-Net. The network was deployed in real-time for the purpose of boundary definition. This boundary was subsequently used for constraint enforcement testing. The robot was utilised to test two different active constraints: a safe region and a restricted region. Results: The network success rate, defined as the ratio of correct over total object localisation results, was calculated to be 54.7% ± 5.2%. In the safe region case, haptic feedback resisted tooltip manipulation beyond the active constraint boundary, with a mean distance from the boundary of 2.70 mm ± 0.37 mm and a mean exit duration of 0.76 s ± 0.11 s. For the restricted-zone constraint, the operator was successfully prevented from penetrating the boundary in 100% of attempts. Discussion: This paper showcases the viability of the proposed robotic platform and presents promising results of a versatile constraint generation and enforcement pipeline.