Low Illumination Research Articles

VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows.

Different from visible cameras which record intensity images frame by frame, the biologically inspired event camera produces a stream of asynchronous and sparse events with much lower latency. In practice, visible cameras can better perceive texture details and slow motion, while event cameras can be free from motion blurs and have a larger dynamic range which enables them to work well under fast motion and low illumination (LI). Therefore, the two sensors can cooperate with each other to achieve more reliable object tracking. In this work, we propose a large-scale Visible-Event benchmark (termed VisEvent) due to the lack of a realistic and scaled dataset for this task. Our dataset consists of 820 video pairs captured under LI, high speed, and background clutter scenarios, and it is divided into a training and a testing subset, each of which contains 500 and 320 videos, respectively. Based on VisEvent, we transform the event flows into event images and construct more than 30 baseline methods by extending current single-modality trackers into dual-modality versions. More importantly, we further build a simple but effective tracking algorithm by proposing a cross-modality transformer, to achieve more effective feature fusion between visible and event data. Extensive experiments on the proposed VisEvent dataset, FE108, COESOT, and two simulated datasets (i.e., OTB-DVS and VOT-DVS), validated the effectiveness of our model. The dataset and source code have been released on: https://github.com/wangxiao5791509/VisEvent_SOT_Benchmark.

UAV Target Tracking Algorithm Based on Illumination Adaptation and Future Awareness in Low Illumination Scenes

Aiming at problems such as tracking failure caused by illumination changes often encountered during unmanned aerial vehicle (UAV) tracking, a target tracking algorithm with illumination adaptive and future-aware correlation filters is proposed based on the background-aware correlation filters (BACF) algorithm, which realizes reliable UAV tracking tasks at night. First, the dark scene is recognized, and an efficient image enhancement module is used to enhance the brightness of low illumination images. Then a future-aware module is constructed to train the tracking model using the contextual information of the target in the next frame for better robustness. Finally, the model updating stage involves adaptive filter updating and adaptive learning rate updating to enhance target tracking precision. The results of the comparison experiments with the state-of-the-art algorithms on the UAVDark135, UAVDT, and DTB70 datasets show that the algorithm in this paper outperforms the state-of-the-art tracking methods and has better tracking performance under light changes and fast motion (FM) scenarios. The tracking speed on a single Central processing unit (CPU) reaches 49 FPS, which satisfies the real-time requirement of UAV tracking.

A low-light image enhancement method for personnel safety monitoring in underground coal mines

Intelligent monitoring technology plays an important role in promoting the development of coal mine safety management. Low illumination in the coal mine underground leads to difficult recognition of monitoring images and poor personnel detection accuracy. To alleviate this problem, a low illuminance image enhancement method proposed for personnel safety monitoring in underground coal mines. Specifically, the local enhancement module maps low illumination to normal illumination at pixel level preserving image details as much as possible. The transformer-based global adjustment module is applied to the locally enhanced images to avoid over-enhancement of bright areas and under-illumination of dark areas, and to prevent possible color deviations in the enhancement process. In addition, a feature similarity loss is proposed to constrain the similarity of target features to avoid the possible detrimental effect of enhancement on detection. Experimental results show that the proposed method improves the detection accuracy by 7.1% on the coal mine underground personal dataset, obtaining the highest accuracy compared to several other methods. The proposed method effectively improves the visualization and detection performance of low-light images, which contributes to the personnel safety monitoring in underground coal mines.

TSRNet: A Trans-Scale and Refined Low-Light Image Enhancement Network

Retinex-based deep learning methods show good low-light enhancement performance and are mainstream approaches in this field. However, the current methods for enhancing low-light images are insufficient in accurately separating illumination and comprehensively restoring degraded information, especially in images with uneven or extremely low illumination levels. This situation often leads to the over-enhancement of bright regions, a loss of detail, and color distortion in the final images. To address these issues, we improved three subnetworks in the classic KinD network, and proposed a trans-scale and refined low-light image enhancement network. Compared with KinD, our method shows more precise image decomposition performance, enhancing the expressiveness of the reflection and illumination components in order to better depict image details, colors, and lighting information. For reflectance restoration, we use a U-shaped network for cross-scale denoising, incorporating attention mechanisms and a color saturation loss to restore image textures and colors. For light adjustment, we apply fine-grained light adjustment approaches to simultaneously enhance brightness in dark areas and prevent excessive enhancement in bright areas. The experimental results demonstrate that with the LOL dataset, the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) of TSRNet are improved by 2–31% and 5–34%, respectively, when compared with the mainstream methods.

Performance Assessment of Cell-Separation Processes for Rear-Contact Solar Cells: Comparison of Indoor and Outdoor Measurements

Half-cell and third-cell modules are the current module technology standard due to their enhanced electrical performance. Also, special module layouts for building or vehicle integrated photovoltaics rely on non-standard cell formats obtained by separation techniques. For either application, interdigitated back-contact (IBC) solar cells are one of the promising cell technologies due to their higher performance and more aesthetic appearance. Thus, a cell separation process, usually employing laser tools, is needed that splits the cells as partial cells are manufactured from full cells. However, this cell cutting also induces additional electrical losses. In this work, we analyze the performance of third cell mini-modules made of IBC cells in comparison to PERC cells as reference. Under standard test conditions (STC), an overall performance gain can only be found for the PERC cells, while current and recombination losses dominate the IBC mini-modules. Outdoor measurements under non-STC conditions reveal reduced performance gains under lower illumination. We show a detailed analysis regarding the performance differences of the IBC and PERC cell technologies at indoor STC and outdoor non-STC test conditions.

Effects of illuminance and correlated color temperature on emotional responses and lighting adjustment behaviors

This article presents a novel approach to assessing human reactions to indoor artificial lighting and lighting adjustment behavior. Participants (n = 27) were each exposed to 17 different lighting conditions via selected combinations of seven illuminance levels and seven different correlated color temperatures (CCTs) in a virtual office environment. We collected data about emotional valence, arousal, adjusted values, and adjustment uncertainty (via the number of button presses) in each lighting condition. We also considered potential gender effects in the responses to the lighting. The study confirmed the presence of “anchor effects,” in which starting conditions significantly impacted subsequent preferences and adjustment behavior. After multiple randomized trials, we also found that lighting preferences shifted toward reduced illuminance. Moreover, women were found to have a lower preference for illumination levels and correlated color temperature than men. A greater Adjustment Uncertainty for illumination was associated with lower illumination values (<290 lx), while this was peaking at mid-range levels for CCT (3500 K–5500 K). The variety of behavioral response profiles we found in lighting adjustment, along with the likely impact of gender and other demographic variables, highlights the importance of customization options in lighting design for higher indoor environmental quality. This research supports architects and engineers in creating lighting designs attuned to occupants' emotions, preferences, and adjustment behaviors while emphasizing the benefits of using VR simulations in the AEC industry to assess lighting design impacts on users.

Research on the Anchor-Rod Recognition and Positioning Method of a Coal-Mine Roadway Based on Image Enhancement and Multiattention Mechanism Fusion-Improved YOLOv7 Model

A drill-anchor robot is an essential means of efficient drilling and anchoring in coal-mine roadways. It is significant to calculate the position of the drill-anchor robot based on the positioning information of the supported anchor rod to improve tunneling efficiency. Therefore, identifying and positioning the supported anchor rod has become a critical problem that needs to be solved urgently. Aiming at the problem that the target in the image is blurred and cannot be accurately identified due to the low and uneven illumination environment, we proposed an improved YOLOv7 (the seventh version of the You Only Look Once) model based on the fusion of image enhancement and multiattention mechanism, and the self-made dataset is used for testing and training. Aiming at the problem that the traditional positioning method cannot guarantee accuracy and efficiency simultaneously, an anchor-rod positioning method using depth image and RGB image alignment combined with least squares linear fitting is proposed, and the positioning accuracy is improved by processing the depth map. The results show that the improved model improves the mAP by 5.7% compared with YOLOv7 and can accurately identify the target. Through the positioning method proposed in this paper, the error between the positioning coordinate and the measurement coordinate of the target point on each axis does not exceed 11 mm, which has high positioning accuracy and improves the positioning accuracy and robustness of the anchor rod in the coal-mine roadway.

Trap-assisted monolayer ReSe2/Si heterojunction with high photoconductive gain and self-driven broadband photodetector.

The development of photodetectors is crucial in fields such as optical communication, image sensing, medical devices and military equipment, where high sensitivity is paramount. We fabricated an ambipolar photodiode using monolayer triclinic ReSe2, synthesized by chemical vapor deposition on p-type Si substrate. The photodetector has a broadband response range from 405 to 1100 nm. The device exhibits high sensitivity to NIR radiation with a high Iph/Idark (ON/OFF) ratio of 5.8 × 104, responsivity (R) of 465 A/W, and specific detectivity (D) of 4.8 × 1013 Jones at open circuit voltage (Voc), indicating photovoltaic behavior. Our ReSe2/Si heterojunction photodetector also exhibits low dark current of 1.4 × 10-9 A and high external quantum efficiency (EQE) of 54368.2% for 1060 nm at -3 V, demonstrating a photoconductive gain. The maximum responsivity (R = 465 A/W) can be achieved at -3 V reverse bias under 1060 nm. The device has a high ideality factor (4.8) and power coefficient (α = 0.5), indicating the presence of interface and sub-gap states that enhance device responsivity at lower illumination intensities by re-exciting trapped carriers into the conduction band. Our results offer important insights into the underlying photo-physics of the ReSe2/Si heterojunction and propose promising avenues for developing advanced broadband photodetectors of high performance.

Abstract B012: Developing applications for a new, fully automated, low-coherence, label free, time lapse holotomographic analysis platform-TomoCube HT-X1: Preparations for AI based analysis

Abstract Our groups are involved in the development of multifunctional nanoparticles and combinations primarily for applications in cancer research is one aspect, we are developing multi-functional for better targeting of the cancer cells. An important part of our efforts is to visualize the effects of the compounds either by visualization, or by quantitative measurements such as the generated by flow cytometry and by laser scanning cytometry. Label free holographic imaging is a new technology that employs laser interference to calculate the refractive index of the pixels. Deconvolution routines create 3-dimensional holograms, and we have developed techniques for 4-dimensional imaging. In Holotomographic imaging there is either a rotating mirror (Nanolive), or a multi-mirror display module (Tomocube) We currently have a Program of Excellence Agreement with Tomocube and are serving as beta testers for their new HT-X1 system. This new instrument abandons laser illumination in favor of low coherence LED illumination. It is a large instrument designed for completely automated operation including autofocus and detector settings, has an automated stage, and is coupled to a high-performance environmental control system. The HT-X1 will have AI capabilities. Some applications. such as accurate cell segmentation and lipid enumeration have already been developed, but have not been released yet. We were told that if we develop our own truth sets, they will be able to process them for us. To illustrate the imaging capabilities, we will use an experiment based on repurposing of an anti-microbial rifamycin as a breast cancer therapeutic. Breast cancer cells were seeded into a 24 well microtiter plate and allowed to attach. Row A was untreated, Row B was treated with free rifamycin, and Row C was treated with liposomal rifamycin. Two-fold dilutions of the drugs were applied starting with column 2. Plates were incubated overnight and imaged on the HT-X1. This experiment showed that at the higher concentration levels, the free rifamycin precipitated leaving rod shaped crystals which presumably reduce its efficacy. The effects of the liposomal rifamycin on the cell morphology is striking. A complete set of the images will be included in the presentation. One of our areas of research is the brain cancer glioblastoma. In previous work using label free tomographic imaging and videography, we identified what we believe to be proto-stem cells, and we made the observation that they are closely associated with nucleoli. In follow-up experiments, we are using a combination of fluorescent dyes, Hoechst 33342 for DNA, Mito tracker green and mitosox red to observe the expression patterns of ROS appearing first in the nucleoli, then spreading to the nucleus, and eventually to the entire cell and the microenvironment. We know that nucleoli are generated from ribosomal proteins at the start of each cell cycle round and disassembled before exiting the G2 phase in a tightly orchestrated sequence. Our hope is that we will be able to further elucidate this using AI. Citation Format: Ed A. Luther, Nina Filipczak, Janaina Artem Ataide, Vladimir P. Torchilin. Developing applications for a new, fully automated, low-coherence, label free, time lapse holotomographic analysis platform-TomoCube HT-X1: Preparations for AI based analysis [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Translating Cancer Evolution and Data Science: The Next Frontier; 2023 Dec 3-6; Boston, Massachusetts. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_2):Abstract nr B012.

Surface Roughness at the Moon’s South Pole: The Influence of Condensed Volatiles on Surface Roughness at the Moon’s South Pole

Condensed volatiles within lunar permanently shadowed regions are of high scientific and resource utilization importance. Volatiles remain elusive and difficult to observe directly, due to low direct solar illumination. In this work, we investigate correlations between, as well as possible effects of, condensed volatiles and surface roughness. We analyze topographic roughness at 50 m and 30–120 m baselines of the lunar south pole (poleward of 85° S). We focus on six south polar craters of interest and their directly surrounding non-cold-trap areas: Faustini, Shoemaker, Haworth, Cabeus, Nobile, and an unnamed region. We further analyze six analogous equatorial craters to investigate the non-ice smoothing contributions: Morozov F, Rosenberger C, Van Maanen, Fraunhofer E, Brisbane, and Asclepi. Lastly, we compare a sunlit and a permanently shaded portion of the Amundsen crater floor. Utilizing data from the Lunar Reconnaissance Orbiter’s Lunar Orbiter Laser Altimeter (LOLA), Lyman Alpha Mapping Project (LAMP), Lunar Reconnaissance Orbiter Camera, and Diviner Lunar Radiometer Experiment, we find subdued roughness within cold traps but determine that roughness is not a unique identifier of condensed volatiles. However, a correlation between LOLA roughness, LAMP normalized Off-band/On-band albedo, temperature, and water-ice stability suggests possible terrain softening due to condensed volatiles, although we cannot rule out dust ponding and/or fairy castle structure contributions. We conclude that LAMP volatile signatures at the topmost ∼100 nm may be indicative of volatile deposits at depths beyond the LAMP sensing capabilities.

Iron-Sensitized Solar Cells (FeSSCs).

ConspectusThe harvesting and conversion of solar energy have become a burning issue for our modern societies seeking to move away from the exploitation of fossil fuels. In this context, dye-sensitized solar cells (DSSCs) have proven to be trustworthy alternatives to silicon-based cells with advantages in terms of transparency and efficiency under low illumination conditions. These devices are highly dependent on the ability of the sensitizer that they contain to collect sunlight and transfer an electron to a semiconductor after excitation. Ruthenium and polypyridine complexes are benchmarks in this field as they exhibit ideal characteristics such as long-lasting metal-ligand charge transfer (MLCT) states and efficient separation between electrons and holes, limiting recombination at the dye-semiconductor interface. Despite all of these advantages, ruthenium is a noble metal, and the development of more sustainable energy devices based on earth-abundant metals is now a must. A quick glance at the periodic table reveals iron as a potential good candidate, since it belongs to the same group of ruthenium, which suggests similar electronic properties. However, striking photophysical differences exist between ruthenium(II) polypyridyl complexes and their Fe(II) analogues, the latter suffering from short-lived MLCT states resulting of their ultrafast relaxation into metal-centered (MC) states. Pyridyl-N-heterocyclic carbenes (pyridylNHC) brought a strong σ-donor character required to promote a higher ligand field splitting of the iron d orbitals. This induces destabilization of the MC states over the MLCT manifold and a consequent slowdown of the excited states deactivation providing iron(II) complexes with tens of picoseconds lifetimes, making them more promising for applications in DSSCs. This Account highlights our recent advances in the development and characterization of iron-sensitized solar cells (FeSSCs) with a focus on the design of efficient sensitizers going from homoleptic to heteroleptic complexes (bearing different anchoring groups) and the tuning of electrolyte composition. Our rational approach led to the best photocurrent and efficiency ever reported for an iron sensitized solar cell (2% PCE and 9 mA/cm2) using a cosensitization process. This work clearly evidences that the solar energy conversion based on iron complex sensitization is now an opened and fruitful route.

Preparation of direct Z-scheme heterojunction la-doped Bi2WO6/CdS with enhanced visible light absorption and its photocatalytic degradation of antibiotics

In this paper, a novel direct Z-scheme heterojunction La-Bi2WO6/CdS composed of 3-dimensional flower-like Bi2WO6 and hexagonally stacked succulent CdS was successfully designed and synthesized. The morphology, optical properties, element valence and crystal structure of the catalysts were systematically characterized. When the dosage of LBC-40 was 1.0 g/L, the degradation rate of oxytetracycline (OTC, 20 mg/L, pH = 6) was 85.28 %, under low visible light illumination (65 W energy saving lamp) for 120 min and the reaction kinetic constant (0.01336 min−1) was 2.50 and 1.56 times that of pure Bi2WO6 and La-Bi2WO6, respectively. Moreover, La-Bi2WO6/CdS showed good photocatalytic activity for other single antibiotics, mixed antibiotics and antibiotics from different water sources. The photocatalytic activity only decreased by 7.69 % after 3 cycles. Furthermore, the possible photocatalytic degradation pathway and photocatalytic mechanism of OTC were proposed. The excellent photocatalytic performance of La-Bi2WO6/CdS is for that the doping of La3+ and the construction of direct Z-scheme heterojunction enhances the visible light absorption capacity and separation efficiency of photogenerated electron-hole pairs. This paper reports a new method, which can provide a reference for the design and development of visible light responsive photocatalysts with excellent photocatalytic properties.

Powering AI at the edge: A robust, memristor-based binarized neural network with near-memory computing and miniaturized solar cell

Memristor-based neural networks provide an exceptional energy-efficient platform for artificial intelligence (AI), presenting the possibility of self-powered operation when paired with energy harvesters. However, most memristor-based networks rely on analog in-memory computing, necessitating a stable and precise power supply, which is incompatible with the inherently unstable and unreliable energy harvesters. In this work, we fabricated a robust binarized neural network comprising 32,768 memristors, powered by a miniature wide-bandgap solar cell optimized for edge applications. Our circuit employs a resilient digital near-memory computing approach, featuring complementarily programmed memristors and logic-in-sense-amplifier. This design eliminates the need for compensation or calibration, operating effectively under diverse conditions. Under high illumination, the circuit achieves inference performance comparable to that of a lab bench power supply. In low illumination scenarios, it remains functional with slightly reduced accuracy, seamlessly transitioning to an approximate computing mode. Through image classification neural network simulations, we demonstrate that misclassified images under low illumination are primarily difficult-to-classify cases. Our approach lays the groundwork for self-powered AI and the creation of intelligent sensors for various applications in health, safety, and environment monitoring.

A novel slime mold algorithm for grayscale and color image contrast enhancement

Image enhancement is a key step in image pre-processing. To address the problem of low quality and visual effect of images under low illumination conditions, this paper proposes an image enhancement method with hyperbolic oscillation factor and quadratic interpolation of slime mold algorithm (SSMA) in non-complete beta function dynamically looking to adjust the grayscale curve. The new strategy mainly proposes three different improvement strategies for the problem that the classical slime mold algorithm has low convergence accuracy and can easily fall into local optimum. Experimenting the proposed SSMA with other conventional and latest algorithms on the CEC2017 benchmark function and low-illumination standard dataset. The experimental results show that the convergence accuracy and convergence speed of SSMA are better than other algorithms. The proposed SSMA-optimized image enhancement algorithm effectively enhances the image brightness while preserving more details of the image, which is significantly better than other algorithms for image enhancement.

Human intrusion detection for high-speed railway perimeter under all-weather condition

PurposeThe paper aims to solve the problem of personnel intrusion identification within the limits of high-speed railways. It adopts the fusion method of millimeter wave radar and camera to improve the accuracy of object recognition in dark and harsh weather conditions.Design/methodology/approachThis paper adopts the fusion strategy of radar and camera linkage to achieve focus amplification of long-distance targets and solves the problem of low illumination by laser light filling of the focus point. In order to improve the recognition effect, this paper adopts the YOLOv8 algorithm for multi-scale target recognition. In addition, for the image distortion caused by bad weather, this paper proposes a linkage and tracking fusion strategy to output the correct alarm results.FindingsSimulated intrusion tests show that the proposed method can effectively detect human intrusion within 0–200 m during the day and night in sunny weather and can achieve more than 80% recognition accuracy for extreme severe weather conditions.Originality/value(1) The authors propose a personnel intrusion monitoring scheme based on the fusion of millimeter wave radar and camera, achieving all-weather intrusion monitoring; (2) The authors propose a new multi-level fusion algorithm based on linkage and tracking to achieve intrusion target monitoring under adverse weather conditions; (3) The authors have conducted a large number of innovative simulation experiments to verify the effectiveness of the method proposed in this article.

A robust autofocusing method for microscopic imaging based on an event camera

Autofocusing is a crucial method for enhancing the efficiency of biological detection workflows. Traditional methods involve calculating various focus metrics on images captured by standard frame-based cameras at multiple static positions to determine the focal point, which can be time-consuming. In contrast, event cameras have the capability to asynchronously detect brightness changes within the field of view of the microscope and provide a continuous stream of events triggered by axial motion. A completely automated coarse-to-fine method is proposed for event based autofocusing, independent of the objective-sample distance. Two criteria for autofocusing in sufficient and low illumination are respectively proposed. The results show that the dynamic accuracy and dynamic repeatability is less than 5 mm.

Nighttime bionic compass based on a short-wave infrared polarization sensing system.

Sky-bionic polar co-ordinate navigation is an effective means of providing navigational information in the absence of a priori information. Polar co-ordinate navigation during clear daytime conditions has been studied, but there has been a lack of research of it at night due to problems with noise. Therefore, in this paper, a short-wave infrared polarimetric sensor system is designed, which is capable of acquiring atmospheric polarimetric information in low illumination environments at night, compared with traditional visible band sensors. Additionally, based on the statistics of polarization angle information, an algorithm for removing noise and starlight is proposed to solve the influence of starlight and noise on the polarization information at night. After many outdoor experiments, we found that the method can output the heading angle stably and accurately, and its standard deviation is controlled to be 0.42° in a clear night.

A Review of Hybrid and Green House Type Solar Dryers

Solar drying is a renewable, efficient, cheap, and sustainable method of preserving agricultural produce. Recent trends in hybrid and greenhouse–type solar dryers were studied. The study revealed that hybrid and greenhouse–type dryers are robust and efficient because they are mostly embedded with heat generating and circulation systems. They are designed to accommodate large-scale drying of fruits and vegetables with higher rates of drying, reduced drying time and some other specific advantages. Findings also revealed that hybrid and greenhouse type solar dryers are mostly designed for optimum retention of heat to compensate for periods with low illumination. The study also gave insight into design challenges peculiar to these types of dryers, and further revealed that most of the fabrications were based on assumptions, with limited data as references. This review also highlights the cost of procurement, uniformity of airflow, sizing of blower and material selection as some of the factors limiting the utilization of hybrid and greenhouse type solar dryers for drying of fruits, vegetables, and other staple crops.

Illumination Distillation Framework for Nighttime Person Re-Identification and a New Benchmark

Nighttime person Re-ID (person re-identification in the nighttime) is a very important and challenging task for visual surveillance but it has not been thoroughly investigated. Under the low illumination condition, the performance of person Re-ID methods usually sharply deteriorates. To address the low illumination challenge in nighttime person Re-ID, this paper proposes an Illumination Distillation Framework (IDF), which utilizes illumination enhancement and illumination distillation schemes to promote the learning of Re-ID models. Specifically, IDF consists of a master branch, an illumination enhancement branch, and an illumination distillation module. The master branch is used to extract the features from a nighttime image. The illumination enhancement branch first estimates an enhanced image from the nighttime image using a nonlinear curve mapping method and then extracts the enhanced features. However, nighttime and enhanced features usually contain data noise due to unstable lighting conditions and enhancement failures. To fully exploit the complementary benefits of nighttime and enhanced features while suppressing data noise, we propose an illumination distillation module. In particular, the illumination distillation module fuses the features from two branches through a bottleneck fusion model and then uses the fused features to guide the learning of both branches in a distillation manner. In addition, we build a real-world nighttime person Re-ID dataset, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Night600</i> , which contains 600 identities captured from different viewpoints and nighttime illumination conditions under complex outdoor environments. Experimental results demonstrate that our IDF can achieve state-of-the-art performance on two nighttime person Re-ID datasets ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Night600</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Knight</i> ). We will release our code and dataset at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Alexadlu/IDF</uri> .

RPM 2.0: RF-Based Pose Machines for Multi-Person 3D Pose Estimation

Advanced human sensing technologies based on radio frequency (RF) signals have gained widespread attention in recent years. However, due to the sparsity and incompleteness of RF signals, fine-grained RF-based multi-person 3D pose estimation has progressed more slowly. In this paper, we present RF-based Pose Machine (RPM 2.0) for multi-person 3D pose estimation using RF signals. Specifically, we first develop a lightweight anchor-free detector module to locate and crop regions of interest from horizontal and vertical RF signals. Afterward, we treat the horizontal and vertical millimeter-wave radars as “RF cameras” with different viewing angles and propose a Multi-view Fusion Network to unproject the RF signals into a unified latent feature space, and then calculate the correlation for weighted fusion. Finally, a Spatio-Temporal Attention Network is designed to reconstruct the multi-person 3D skeleton sequences, in which the spatial attention module is proposed to recover invisible body parts using non-local correlations among joints and the temporal attention module refines the 3D pose sequences using temporal coherency learned from frame queries. We evaluate the performance of the proposed RPM 2.0 and state-of-the-art methods on a large-scale dataset with multi-person 3D pose labels and corresponding radar signals. The experimental results show that RPM 2.0 outperforms all of the baseline methods, which locates multi-person 3D key points with an average error of 73 <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mm</i> and generalizes well in new data such as occlusion, low illumination.