Temporal Locality Research Articles

Action-Aware Network with Upper and Lower Limit Loss for Weakly-Supervised Temporal Action Localization

Action-Aware Network with Upper and Lower Limit Loss for Weakly-Supervised Temporal Action Localization

A Rapid Temporal Bone Localization Method Based on Machine Visual Detection Markers

The number of hearing-impaired people is increasing year by year; robotic cochlear drilling surgery is one of the safest methods to treat deafness. Looking at the issue of low efficiency of temporal bone posture positioning in cochlear implantation robotic drilling, a novel auxiliary ring marker temporal bone positioning method was proposed to improve temporal bone posture positioning efficiency, optimize the operation time, and reduce auxiliary injuries caused by the surgery. First, the temporal bone visual positioning assistant ring was designed based on the requirements for cochlear robotic drilling surgery. The target detection was conducted on the auxiliary ring and image processing and feature point extraction methods were designed. Then, the three-dimensional coordinates of the measured feature points were obtained by binocular vision, and the auxiliary ring and temporal bone postures were estimated. Finally, the auxiliary ring and temporal bone localization methods were validated. The experiment results indicated that the temporal bone was located quickly and effectively in a total time of about 33 ms, which was faster and more accurate than traditional visual localization methods and could satisfy real-time temporal bone localization during surgery. This study can reduce the time of temporal bone visual positioning in cochlear implant drilling operations, greatly improving the robot’s capabilities to extract visual information during the operation, which has a better auxiliary role for future research and applications of the cochlear implant drilling operation.

OW-TAL: Learning Unknown Human Activities for Open-World Temporal Action Localization

Current temporal action localization methods work well on a closed-world assumption, in which all action categories to be localized are known as a priori. However, this assumption doesn’t apply to open-world scenarios, as novel categories that never appeared in the training stage will be encountered without explicit supervision. Distinct from the closed-world setting, localizing actions under the open-world setup poses two significant challenges: 1) identifying unknown actions from diverse knowns and localizing their temporal boundaries. 2) defying forgetting of previous actions when incrementally updating knowledge of identified unknown actions. To address the aforementioned challenges, we develop a two-branch framework with Unknown and Known action modeling Networks, a.k.a. UK-Net, for the problem of Open-World Temporal Action Localization (OW-TAL). The potential patterns underlying unknown and known actions, as well as their dynamic transformation, are modeled in a unified pipeline. Specifically, a self-attention based position-sensitive module is designed to produce actionness scores for unknown actions in a class-agnostic way. Besides, an iterative optimization strategy is developed to enable knowledge derived from known categories to be shared with the unknowns. In addition, a self-paced learning strategy is proposed to instructionally guide class-incremental learning while defying catastrophic forgetting. Benefiting from the above components, our UK-Net yields superior performance on three challenging datasets, i.e., THUMOS14, ActivityNet1.2, and MUSES. Experimental results also demonstrate the competitive performance of our method when compared with traditional closed-world counterparts.

Temporal Moment Localization via Natural Language by Utilizing Video Question Answers as a Special Variant and Bypassing NLP for Corpora

Temporal moment localization using natural language (TMLNL) is an emerging issue in computer vision for localizing a specific moment inside a long, untrimmed video. The goal of TMLNL is to obtain the video’s output moment, which is related to the input query in a substantial way. Previous research focused on the visual portion of TMLNL, such as objects, backdrops, and other visual attributes, but natural language processing (NLP) techniques were largely used for the textual portion. A long query requires sufficient context to properly localize moments within a long untrimmed video. Thus, as a consequence of not completely understanding how to handle queries, performances deteriorated, especially when the query was longer. In this paper, we treat the TMLNL challenge as a unique variation of VQA, which equally considers the visual elements by using our proposed VQA joint visual-textual framework (JVTF). However, we also manage complex and long input queries without employing natural language processing (NLP) by improving poorly graded to finely graded distinct granularity representations. Our suggested BCPN searches for insufficient context for long input queries using an approach called query handler (QH) and helps the JVTF find the most relevant moment. Previously, a recurrence of words was caused by increasing the number of encoding layers in transformers, LSTMs, and other NLP techniques; however, our QH ensured that repetition of word locations was reduced. The output of BCPN is combined with JVTF’s guided attention to further improve the end outcome. Therefore, we propose a novel bidirectional context predictor network (BCPN), in addition to a VQA joint visual-textual framework (JVTF), to address the equal importance of videos and queries. Through extensive experiments on three benchmark datasets, we show that the proposed BCPN outperforms the state-of-the-art methods by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.3 (2.65 \%) $ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.5 (2.49 \%)$ </tex-math></inline-formula> , and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$IoU = 0.7 (2.06 \%) $ </tex-math></inline-formula> .

Neuromorphic Vision-Based Fall Localization in Event Streams With Temporal-Spatial Attention Weighted Network.

Falling down is a serious problem for health and has become one of the major etiologies of accidental death for the elderly living alone. In recent years, many efforts have been paid to fall recognition based on wearable sensors or standard vision sensors. However, the prior methods have the risk of privacy leaks, and almost all these methods are based on video clips, which cannot localize where the falls occurred in long videos. For these reasons, in this article, the bioinspired vision sensor-based falls temporal localization framework is proposed. The bioinspired vision sensors, such as dynamic and active-pixel vision sensor (DAVIS) camera applied in this work responds to pixels' brightness change, and each pixel works independently and asynchronously compared to the standard vision sensors. This property makes it have a very high dynamic range and privacy preserving. First, to better represent event data, compared with the typical constant temporal window mechanism, an adaptive temporal window conversion mechanism is developed. The temporal localization framework follows a proven proposal and classification paradigm. Second, for the high-efficient and recall proposal generation, different from the traditional sliding window scheme, the event temporal density as the actionness score is set and the 1D-watershed algorithm to generate proposals is applied. In addition, we combine the temporal and spatial attention mechanism with our feature extraction network to temporally model the falls. Finally, to evaluate the performance of our framework, 30 volunteers are recruited to join the simulated fall experiments. According to the results of experiments, our framework can realize precise falls temporal localization and achieve the state-of-the-art performance.

Frame-rate up-conversion detection based on convolutional neural network for learning spatiotemporal features

With the advance in user-friendly and powerful video editing tools, anyone can easily manipulate videos without leaving prominent visual traces. Frame-rate up-conversion (FRUC), a representative temporal-domain operation, increases the motion continuity of videos with a lower frame-rate and is used by malicious counterfeiters in video tampering such as generating fake frame-rate video without improving the quality or mixing temporally spliced videos. FRUC is based on frame interpolation schemes and subtle artifacts that remain in interpolated frames are often difficult to distinguish. Hence, detecting such forgery traces is a critical issue in video forensics. This paper proposes a frame-rate conversion detection network (FCDNet) that learns forensic features caused by FRUC in an end-to-end fashion. The proposed network uses a stack of consecutive frames as the input and effectively learns interpolation artifacts using network blocks to learn spatiotemporal features. Moreover, it can cover the following three types of frame interpolation schemes: nearest neighbor interpolation, bilinear interpolation, and motion-compensated interpolation. In contrast to existing methods that exploit all frames to verify integrity, the proposed approach achieves a high detection speed because it observes only six frames to test its authenticity. Extensive experiments were conducted with conventional forensic methods and neural networks for video forensics to validate our research. The proposed work achieved an outstanding performance in terms of detecting the interpolated artifacts of FRUC. The experimental results also demonstrate that our model is robust against an unseen dataset, unlearned frame-rate, and unlearned quality factor. Furthermore, FCDNet can precisely localize the tampered region applied to manipulation along the time-domain through temporal localization.

Between Hard and Soft Power: British and Russian Youth on National Pride

AbstractThis study examines the collective memory of British and Russian youth. We used the results of a comparative survey conducted among Russian and British students. The study focuses primarily on pride in the collective memory of young people with the aim of analyzing the category of pride among young people across several dimensions. First, we look at the qualitative content of national pride: pride in the realization of tasks related to “soft power” (for example, culture, education, sports), and pride in manifestations of “hard power” (for example, pride in military victories or power politics). Second, we analyze the temporal localization of national pride: where are the main events, personalities, and phenomena study participants take pride in, both in the past and in the present. Third, an important element of understanding pride in a country is the relationship between pride and shame: what events are mentioned more often: shameful or pride-inspiring.

GLFormer: Global and Local Context Aggregation Network for Temporal Action Detection

As the core component of video analysis, Temporal Action Localization (TAL) has experienced remarkable success. However, some issues are not well addressed. First, most of the existing methods process the local context individually, without explicitly exploiting the relations between features in an action instance as a whole. Second, the duration of different actions varies widely; thus, it is difficult to choose the proper temporal receptive field. To address these issues, this paper proposes a novel network, GLFormer, which can aggregate short, medium, and long temporal contexts. Our method consists of three independent branches with different ranges of attention, and these features are then concatenated along the temporal dimension to obtain richer features. One is multi-scale local convolution (MLC), which consists of multiple 1D convolutions with varying kernel sizes to capture the multi-scale context information. Another is window self-attention (WSA), which tries to explore the relationship between features within the window range. The last is global attention (GA), which is used to establish long-range dependencies across the full sequence. Moreover, we design a feature pyramid structure to be compatible with action instances of various durations. GLFormer achieves state-of-the-art performance on two challenging video benchmarks, THUMOS14 and ActivityNet 1.3. Our performance is 67.2% and 54.5% AP@0.5 on the datasets THUMOS14 and ActivityNet 1.3, respectively.

PcmNet: Position-sensitive context modeling network for temporal action localization

Temporal action localization, which aims to locate temporal regions where actions take place and recognize their corresponding classes in untrimmed real-world videos, is a challenging task. As a critical cue to video understanding, exploiting the video context has become an important strategy to boost the localization performance. However, previous methods mainly focus on exploring semantic context which captures the feature similarity among frames or proposals. The temporal position context which is also vital for temporal action localization is less explored. In this paper, we propose a position-sensitive context modeling approach to fuse both semantic and position context for more precise action localization. Specifically, we first propose a position encoding method tailored for temporal action localization on both frame-level and proposal-level, which ensures that the generated position representations can model the distance and chronological relationships among frames or proposals. Then we conduct attention-based context aggregation to produce discriminative features and help with precise boundary detection and proposal evaluation. Our method achieves state-of-the-art performance on two widely used datasets, THUMOS-14 and ActivityNet-1.3, demonstrating the effectiveness and generalizability of our method.

A Study on Features of Different Tone Quality in a Kenong Set

This work discusses how to distinguish kenong frequencies in a signal and the time-localized frequency content for each tone at a given time using an audio-based approach to tuning retrieval where the fundamental and overtone pitch is shown at all frequencies at a given time. The method of temporal localization on the dominant frequency at its unique time for each tone allows for the detection of frequencies present in the signal. Two approaches used in retrieving the harmonic, pitch, and timbre of kenong are Picoscope and Melda analyzer. The audio recording was done using an At4050 microphone and Ur22 audio interface in mono at 24-bit resolution and 48 kHz sampling rate. PicoScope produces the spectrum while the Melda analyzer produces changes in the spectra with time. Kenong D, E, G, A, and C displayed their near overtones at (2:2.8:4.0), (2:3.0:3.9), (2:2.9:3.9), (2:2.6:3.9), and (2:2.6:3.9). Kenong D had a strong fundamental peak at 295Hz. Kenong G keeps the fundamental frequency constant until t=5s. The basic peak was maintained by Kenong C. The results reveal that the kenong was properly tuned, although the tuner solely tuned it based on hearing, passed down from generation to generation. The maker’s intuition permits him to create a specific ‘signature’ through sound unique to a given kenong set.

Weakly supervised temporal action localization with proxy metric modeling

Weakly supervised temporal action localization with proxy metric modeling

Dual Locality-Based Flash Translation Layer for NAND Flash-Based Consumer Electronics

NAND flash memory shows prominent performance, so it has been used as storage devices of consumer electronics, such as the smart phones and tablet personal computers. As the storage management software of NAND flash memory, the page-level flash translation layer (PLFTL) owns very high I/O access performance for consumer electronics. As an improved version of PLFTL, the demand-based PLFTL selectively keeps active mapping entries in the DRAM (Dynamic Random Access Memory) and the demand-based PLFTL mainly considers the temporal locality of workloads. However, the spatial locality also appears in many workloads. To exploit the temporal locality and spatial locality of workloads, a novel dual locality-based FTL (DL-FTL) is proposed in this paper. DL-FTL uses the sequential cache mapping state table (S-CMST) and sequential physical address cache mapping table (SPA-CMT) to process the sequential requests. To decrease the update counts of translation pages, the mapping entries that are evicted from S-CMST will be written back to NAND flash memory using a batch update strategy. The experimental results show that our proposed DL-FTL raises the cache hit ratio by up to 66.39% and reduces the system response time by up to 21.64% on average, compared with the demand-based PLFTL.

HCFTL: A Locality-Aware Flash Translation Layer for Efficient Address Translation

In modern solid-state drives (SSDs), a flash translation layer (FTL) is the core engine performing the logical-to-physical address translations. However, the increasing capacity of SSDs requires a large DRAM space to hold the mapping table in a page-level FTL. Due to the limited size of built-in DRAM, existing FTL schemes selectively cache some active mapping entries in DRAM, while store the entire mapping table on flash. However, the low cache hit ratio introduced by the traditional mapping entry eviction policy degrades the access performance of SSDs. In this article, we propose a novel FTL, hot-clusterity FTL (HCFTL), to improve the hit ratio of cached mapping table (CMT) of SSDs with limited cache space. HCFTL clusters mapping entries, which are newly evicted from the cache, into dynamic translation pages (DTPs). Reading entries from DTPs will increase the CMT hit ratio due to spatial and temporal localities. Because the logical page numbers (LPNs) in a DTP are not consecutive, we introduce two different efficient index structures to speedup the lookup of mapping entries in DTPs. Furthermore, to make the DTP index highly accurate, we additionally allocate an auxiliary cache to buffer the mapping entries newly evicted from CMT, and produce dynamic pages with minimum difference between the maximal and the minimal LPNs of entries in those pages. Our experiments show that HCFTL can improve the CMT hit ratio by up to 41.1% and decrease the system response time by up to 33.3%, compared to baseline FTL schemes.

Weakly-Supervised Action Localization, and Action Recognition Using Global–Local Attention of 3D CNN

3D convolutional neural network (3D CNN) captures spatial and temporal information on 3D data such as video sequences. However, due to the convolution and pooling mechanism, the information loss that occurs seems unavoidable. To improve the visual explanations and classification in 3D CNN, we propose two approaches; (i) aggregate layer-wise global to local (global–local) discrete gradient using trained 3DResNext network, and (ii) implement attention gating network to improve the accuracy of the action recognition. The proposed approach intends to show the usefulness of every layer termed as global–local attention in 3D CNN via visual attribution, weakly-supervised action localization, and action recognition. Firstly, the 3DResNext is trained and applied for action classification using backpropagation concerning the maximum predicted class. The gradient and activation of every layer are then up-sampled. Later, aggregation is used to produce more nuanced attention, which points out the most critical part of the predicted class’s input videos. We use contour thresholding of final attention for final localization. We evaluate spatial and temporal action localization in trimmed videos using fine-grained visual explanation via 3DCAM. Experimental results show that the proposed approach produces informative visual explanations and discriminative attention. Furthermore, the action recognition via attention gating of each layer produces better classification results than the baseline model.

Progressive enhancement network with pseudo labels for weakly supervised temporal action localization

Weakly supervised temporal action localization (WSTAL) is crucial for real world applications, as it relieves the huge burden of frame-level annotations for fully supervised action detection. Most existing WSTAL methods focused on classifying video snippets, or detecting action boundaries. However, the predictions from these well-designed models have not been fully utilized. Accordingly, we propose a weakly-supervised framework called the progressive enhancement network (PEN), which takes full advantages of the predictions generated by the preceding models to enhance the subsequent models. Specifically, snippet-level pseudo labels are generated from the preceding predictions by considering the similarity and temporal distance between action snippets. Then subsequent models are progressively enhanced by using pseudo labels as a supervision, and utilizing their underlying semantics to make the feature representation more qualified for the temporal localization task. Extensive experiments which are carried out on two popular benchmarks, THUMOS’14 and ActivityNet v1.2, demonstrate the effectiveness of our method.

SwNEMO_v4.0: an ocean model based on NEMO4 for the new-generation Sunway supercomputer

Abstract. The current large-scale parallel barrier of ocean general circulation models (OGCMs) makes it difficult to meet the computing demand of high resolution. Fully considering both the computational characteristics of OGCMs and the heterogeneous many-core architecture of the new Sunway supercomputer, swNEMO_v4.0, based on NEMO4 (Nucleus for European Modelling of the Ocean version 4), is developed with ultrahigh scalability. Three innovations and breakthroughs are shown in our work: (1) a highly adaptive, efficient four-level parallelization framework for OGCMs is proposed to release a new level of parallelism along the compute-dependency column dimension. (2) A many-core optimization method using blocking by remote memory access (RMA) and a dynamic cache scheduling strategy is applied, effectively utilizing the temporal and spatial locality of data. The test shows that the actual direct memory access (DMA) bandwidth is greater than 90 % of the ideal bandwidth after optimization, and the maximum is up to 95 %. (3) A mixed-precision optimization method with half, single and double precision is explored, which can effectively improve the computation performance while maintaining the simulated accuracy of OGCMs. The results demonstrate that swNEMO_v4.0 has ultrahigh scalability, achieving up to 99.29 % parallel efficiency with a resolution of 500 m using 27 988 480 cores, reaching the peak performance with 1.97 PFLOPS.

Sampling Techniques and Error Estimation for Linear Canonical S Transform Using MRA Approach

A linear canonical S transform (LCST) is considered a generalization of the Stockwell transform (ST). It analyzes signals and has multi-angle, multi-scale, multiresolution, and temporal localization abilities. The LCST is mostly suitable to deal with chirp-like signals. It aims to possess the characteristics lacking in a classical transform. Our aim in this paper was to derive the sampling theorem for the LCST with the help of a multiresolution analysis (MRA) approach. Moreover, we discuss the truncation and aliasing errors for the proposed sampling theory. These types of sampling results, as well as methodologies for solving them, have applications in a wide range of fields where symmetry is crucial.

BreathMentor

Chronic Obstructive Pulmonary Disease (COPD) is currently the third major cause of death--more than three million people died from it in 2019. Given that COPD cannot be cured currently, immediate treatment is crucial. Pulmonary rehabilitation (PR) is widely used to prevent COPD deterioration. Patients are advised to undergo a PR at home to get sufficient treatment in time. Monitoring patients during home rehabilitation can help not only improve patient adherence but also collect data on patients' recovery progress from rehabilitation team's perspective. However, how to track if proper diaphragmatic breathing, an essential part of PR, is taken by a patient has remained challenging. The current monitoring solution still appears obtrusive as it requires the patient to wear two uncomfortable respiration belts. Alternatively, therapists need to monitor the patients remotely through several cameras, which consumes substantial medical resources and causes privacy issues. In this work, we present BreathMentor, a smart speaker based diaphragmatic breathing monitoring system targeting early COPD stages I and II. BreathMentor is both unobtrusive and preventive of privacy invasion, so that it can solve the existing pain points and suits home care. BreathMentor converts the smart speaker into an active sonar system that continuously perceives and analyses the changes in surroundings, thereby detecting the user's respiration rate, deriving the breathing phases, and classifying whether the patient is practising diaphragmatic breathing. BreathMentor formulates breathing monitoring as a Temporal Action Localization task that enables us to detect each breathing cycle and classify its type. Our key insight is that breathing periodicity and phase duration are natural properties to localize and segment the breaths. Our key design to classify the breathing type is a hybrid architecture encompassing signal processing and deep learning techniques. Further, we evaluate the system performance on fifteen healthy subjects who would not breathe abnormally during diaphragmatic breathing under the supervision of therapists. In conclusion, BreathMentor can achieve robust performance for monitoring diaphragmatic breathing in different environments, as demonstrated in the results. The median error rate of respiration detection is 0.2 BPM, and the I/E ratio derivation is accurate with a mean absolute percentage error of less than 5.9% for breathing phase detection, together with a recall of 98.2%, and a precision of 95.5% in detecting diaphragmatic breathing. Above results indicate that BreathMentor can be used to track the patients' adherence and help monitor their breathing capacity.

THE USE OF MARKOV ALGORITHMS FOR THE STUDY OF l-VOIDS IN BIG DATA OF SOCIO-ECONOMIC SYSTEMS. PART 2

The second part of the article is presented. Big data l-voids are considered using the N-scheme of the Markov algorithm. The diagrams of occurrences of l-voids in a semi-Eulerian cycle containing an Euler path, a matroid and an incomplete Fano matroid, minors K3, 3 and K5, an extra large cycle of occurrences are analyzed. An example of reconstructing a fragment of an incomplete Fano matroid with l-voids is considered. Examples are given for independent implementation of the method of filling the artificial intelligence database (AnwM f typeK) DB based on the results of the analysis of l-voids in the N-scheme of the Markov algorithm. The database is populated using a formal language with the alphabet M = litj/abdgckm where litj are terminal elements, abdgckm are non-terminal elements, and the basis of the alphabet is i – initial, t – terminal, j – isomorphic, l-empty morphisms. The p – initial, e – simple, and h – final occurrences of words from the Markov alphabet A are determined. An example of a simplified numbering of occurrence chains is given, which ensures the selection of all objects represented by words in the Markov alphabet A, belonging to “their own” chain of occurrences, as well as all related to chain object. The mechanism of dynamic change of the number-type in the N-scheme of the Markov algorithm is presented, which provides the analysis of the word as an evolving category, as well as the evolution of categories, which includes such a word. It has been established that normal Markov inference is characterized by temporal conditionality and locality. The conclusion is made about the expediency of considering the N-scheme in a distributed computing system. In this case, it is necessary to analyze the diagrams of occurrences of objects that have several designations with words from the Markov alphabet A, in order to identify them and process data from such objects.

Learning Disentangled Classification and Localization Representations for Temporal Action Localization

A common approach to Temporal Action Localization (TAL) is to generate action proposals and then perform action classification and localization on them. For each proposal, existing methods universally use a shared proposal-level representation for both tasks. However, our analysis indicates that this shared representation focuses on the most discriminative frames for classification, e.g., ``take-offs" rather than ``run-ups" in distinguishing ``high jump" and ``long jump", while frames most relevant to localization, such as the start and end frames of an action, are largely ignored. In other words, such a shared representation can not simultaneously handle both classification and localization tasks well, and it makes precise TAL difficult. To address this challenge, this paper disentangles the shared representation into classification and localization representations. The disentangled classification representation focuses on the most discriminative frames, and the disentangled localization representation focuses on the action phase as well as the action start and end. Our model could be divided into two sub-networks, i.e., the disentanglement network and the context-based aggregation network. The disentanglement network is an autoencoder to learn orthogonal hidden variables of classification and localization. The context-based aggregation network aggregates the classification and localization representations by modeling local and global contexts. We evaluate our proposed method on two popular benchmarks for TAL, which outperforms all state-of-the-art methods.