Scene Analysis Research Articles

The effect of angular disparity between sound source sequences on the perceptual organization of auditory objects

Within auditory scene analysis is a well-known streaming phenomenon which is observed when listeners group several sounds into “auditory objects” in complex auditory scenes. When two auditory sequences consist of similar sounds, these sequences tend to be perceived as a single stream, but can form segregated streams when the sounds are presented from disparate angles. This study investigated the tolerance thresholds for the angular disparity at which the sound sources would segregate rather than group together into single auditory objects, when 50-ms one-third octave bands of noise were presented in the horizontal plane with changing azimuth angle. Listeners were asked to judge whether the presented noise sequences were perceived as a single stream or not. The results revealed that listeners tend to perceive the presented sequences as a single stream when the angular disparity between noises was small; however, these tolerance thresholds depended on the spatial region within which the sequences were presented. Tolerance thresholds became large when the noise sequences were presented from the side, in comparison to the situation when sequences were presented from in front of the listeners. Results were consistent with the minimum audible angle on the horizontal plane.

CogniWatch: Advanced Scene Analysis for Threat Detection and Public Safety

Ensuring the safety of life and property through the deployment of high-quality CCTV cameras has become indispensable in our modern world. Manual monitoring of every moment is not feasible, and the unpredictable nature of human behavior makes distinguishing between suspicious and normal activities a formidable challenge. In this research, we introduce a novel approach utilizing Convolutional Neural Networks (CNN) to discern between suspicious and routine activities within an environment. Our proposed system is designed to automatically alert relevant authorities upon detecting potentially suspicious behavior. The effectiveness of any suspicious activity detection system hinges on several critical factors, including the quality of training data, the architecture of the Machine Learning model, and the operational environment. To maintain the system's accuracy and keep it adaptive to new and evolving threats, continuous monitoring, regular updates, and ongoing improvement are imperative. Our work underscores the importance of robust data sources and the careful design of CNN-based models to ensure the system's reliability in real-world applications. This research not only addresses the pressing need for automated surveillance but also emphasizes the significance of staying current and vigilant in the ever- changing landscape of safety and security. KeyWords: CNN, Object Detection, Anomaly, Threat, Hyperparameter Tunning.

MT-FiST: A Multi-Task Fine-grained Spatial-Temporal Framework for Surgical Action Triplet Recognition.

Surgical action triplet recognition plays a significant role in helping surgeons facilitate scene analysis and decision-making in computer-assisted surgeries. Compared to traditional context-aware tasks such as phase recognition, surgical action triplets, comprising the instrument, verb, and target, can offer more comprehensive and detailed information. However, current triplet recognition methods fall short in distinguishing the fine-grained subclasses and disregard temporal correlation in action triplets. In this paper, we propose a multi-task fine-grained spatial-temporal framework for surgical action triplet recognition named MT-FiST. The proposed method utilizes a multi-label mutual channel loss, which consists of diversity and discriminative components. This loss function decouples global task features into class-aligned features, enabling the learning of more local details from the surgical scene. The proposed framework utilizes partial shared-parameters LSTM units to capture temporal correlations between adjacent frames. We conducted experiments on the CholecT50 dataset proposed in the MICCAI 2021 Surgical Action Triplet Recognition Challenge. Our framework is evaluated on the private test set of the challenge to ensure fair comparisons. Our model apparently outperformed state-of-the-art models in instrument, verb, target, and action triplet recognition tasks, with mAPs of 82.1% (+4.6%), 51.5% (+4.0%), 45.50% (+7.8%), and 35.8% (+3.1%), respectively. The proposed MT-FiST boosts the recognition of surgical action triplets in a context-aware surgical assistant system, further solving multi-task recognition by effective temporal aggregation and fine-grained features.

Effects of age-related cochlear deafferentation and central gain on auditory scene analysis

Animal models show that cochlear afferent nerve endings are more vulnerable than sensory hair cells to age-related damage. Because such cochlear deafferentation is not apparent in standard audiometry, the extent to which it contributes to deficits in human hearing is debated, and the intervening neural processes are poorly characterized. This presentation will describe our efforts to address these gaps through co-ordinated experiments in at-risk humans and a chinchilla model. Our results suggest that cochlear deafferentation is widespread in middle age despite clinically normal audiometric sensitivity. Furthermore, the resulting reduction in peripheral input appears to trigger compensatory central “gain” at the cortical level, likely through altered local balance of excitation and inhibition. Consistent with the important role of inhibition in parsing temporal regularities across different frequency components of sound, central gain is also associated with reduced ability to perceptually segregate acoustic scenes with multiple sources into individual perceptual streams. Taken together, our results suggest that age-related cochlear deafferentation may affect hearing not only by reducing the fidelity of input encoding but also by interfering with the central auditory system’s ability to extract targets in noisy environments.

Interactions between attention and auditory scene analysis

Do auditory objects form automatically, or only when a listener focuses attention on a sound? This review argues that the question is ill posed—that both are simultaneously correct. Intuition suggests that the processes of source segregation and attention happen sequentially: first segregation parses a complex scene into constituent objects, and then selection pulls out an important sound to be analyzed in detail. But, rather than a processing hierarchy in which object formation occurs first, followed by selection, auditory scene analysis and attention interact in a heterarchy: formation and selection influence one another, feed back upon each other, and are not easily separable in terms of either how they are implemented in the brain or how their effects can be measured. There is not one particular site in the auditory pathway where objects “first appear;” instead, an object-based representation emerges gradually and imperfectly from the auditory nerve through the brainstem and midbrain to the various divisions of the cortex. Similarly, attentional selection accrues at every stage as one moves from the ear to the cortex. This talk will review examples from perceptual and physiological experiments supporting the idea that attention and object formation interact heterarchical.

Analysis of the fusion development of traditional culture concept and interior art design based on 3D scene analysis

Abstract In this paper, by collecting a large number of 3D scenes with traditional cultural styles, after pre-processing, the style analysis task of 3D scenes is realized through two stages of model style element discovery and scene style pattern mining. In the style pattern mining stage, the maximum frequent pattern mining algorithm is used to get the style pattern of the final scene, and the part is visualized. To reduce the user’s selection time, a 3D interior layout recommendation system based on style rule mining is designed using collaborative filtering. Based on the previous work on scene style pattern mining, the style association rules are further mined, and the user is provided with candidate layouts and furniture dynamically and in real-time by introducing user interaction. Comparing the text-improved recommendation algorithm with the traditional recommendation algorithm, the accuracy rate of the traditional algorithm starts to saturate to 65% after TopN>20, while the algorithm in this paper is higher than 80%. In the analysis of traditional cultural contexts of interior art, in terms of the aesthetic acquisition, the highest proportion of aesthetic association is 63.7%. In terms of emotional expression, the proportion of symbolization of emotion is 73.1%. In terms of object comparison, the highest proportion of plurality and flexibility of combinations was 57.5%. Therefore, this paper provides good research ideas for how to better integrate traditional culture into interior design.

Detection and Classification of Buildings by Height from Single Urban High-Resolution Remote Sensing Images

Recent improvements in remote sensing technologies have boosted building detection techniques from rough classifications using moderate resolution imagery to precise extraction from high-resolution imagery. Shadows frequently emerge in high-resolution urban images. To exploit shadow information, we developed a novel building detection and classification algorithm for images of urban areas with large-size shadows, employing only the visible spectral bands to determine the height levels of buildings. The proposed method, building general-classified by height (BGCH), calculates shadow orientation, detects buildings using seed-blocks, and classifies the buildings into different height groups. Our proposed approach was tested on complex urban scenes from Toronto and Beijing. The experimental results illustrate that our proposed method accurately and efficiently detects and classifies buildings by their height levels; the building detection rate exceeded 95%. The precision of classification by height levels was over 90%. This novel building-height-level detection method provides rich information at low cost and is suitable for further city scene analysis, flood disaster risk assessment, population estimation, and building change detection applications.

The Interplay of Dominant Empiricism and Residual Cartesianism in Ann Radcliffe’s A Sicilian Romance

ABSTRACT In response to the supernatural occurrences in the Marquis’s castle, several characters of Ann Radcliffe’s A Sicilian Romance adhere to the tenets of Lockean empiricism to interpret their sensory perception. Conversely, a few characters adopt Cartesian scepticism to dismiss the validity of sensory experience. To date, no exhaustive study of Ann Radcliffe’s fiction has inspected the incorporation of Lockean empiricism and the Cartesian method of René Descartes into her novels. By conducting a historical analysis of the philosophical scene of the Enlightenment era, this paper aims to substantiate that not only does A Sicilian Romance encompass seminal philosophical notions of the age of Enlightenment, but it also points to the shifting philosophical scene within this period, characterised by the dominance of empiricism and the residuality of Cartesianism. By examining the reflection of these philosophies in Radcliffe’s novel, this article challenges the supposition that the Gothic novel necessarily contains anti-Enlightenment impulses.

Advances in Data‐Driven Analysis and Synthesis of 3D Indoor Scenes

AbstractThis report surveys advances in deep learning‐based modelling techniques that address four different 3D indoor scene analysis tasks, as well as synthesis of 3D indoor scenes. We describe different kinds of representations for indoor scenes, various indoor scene datasets available for research in the aforementioned areas, and discuss notable works employing machine learning models for such scene modelling tasks based on these representations. Specifically, we focus on the analysis and synthesis of 3D indoor scenes. With respect to analysis, we focus on four basic scene understanding tasks – 3D object detection, 3D scene segmentation, 3D scene reconstruction and 3D scene similarity. And for synthesis, we mainly discuss neural scene synthesis works, though also highlighting model‐driven methods that allow for human‐centric, progressive scene synthesis. We identify the challenges involved in modelling scenes for these tasks and the kind of machinery that needs to be developed to adapt to the data representation, and the task setting in general. For each of these tasks, we provide a comprehensive summary of the state‐of‐the‐art works across different axes such as the choice of data representation, backbone, evaluation metric, input, output and so on, providing an organized review of the literature. Towards the end, we discuss some interesting research directions that have the potential to make a direct impact on the way users interact and engage with these virtual scene models, making them an integral part of the metaverse.

Neural dynamics underlying successful auditory short-term memory performance.

Listeners often operate in complex acoustic environments, consisting of many concurrent sounds. Accurately encoding and maintaining such auditory objects in short-term memory is crucial for communication and scene analysis. Yet, the neural underpinnings of successful auditory short-term memory (ASTM) performance are currently not well understood. To elucidate this issue, we presented a novel, challenging auditory delayed match-to-sample task while recording MEG. Human participants listened to 'scenes' comprising three concurrent tone pip streams. The task was to indicate, after a delay, whether a probe stream was present in the just-heard scene. We present three key findings: First, behavioural performance revealed faster responses in correct versus incorrect trials as well as in 'probe present' versus 'probe absent' trials, consistent with ASTM search. Second, successful compared with unsuccessful ASTM performance was associated with a significant enhancement of event-related fields and oscillatory activity in the theta, alpha and beta frequency ranges. This extends previous findings of an overall increase of persistent activity during short-term memory performance. Third, using distributed source modelling, we found these effects to be confined mostly to sensory areas during encoding, presumably related to ASTM contents per se. Parietal and frontal sources then became relevant during the maintenance stage, indicating that effective STM operation also relies on ongoing inhibitory processes suppressing task-irrelevant information. In summary, our results deliver a detailed account of the neural patterns that differentiate successful from unsuccessful ASTM performance in the context of a complex, multi-object auditory scene.

REPRESENTASI FEMINISME DALAM IKLAN KECAP ABC “SUAMI SEJATI MAU MASAK”

The kecap ABC advertisement for the suami Sejati Mau Masak edition is a series in the gender equality campaign created by Heinz ABC Indonesia. Through this series of advertisements, it is made as a form of appreciation to women who have done domestic work, especially cooking. In this ad, many messages of feminism are shown visually in every scene. The purpose of this research is to analyze how the message of feminism is contained in the Ketchup ABC advertisement of “suami sejati mau masak” using Rolnad Barthes's semiotic theory approach. In this study, it was carried out using descriptive qualitative research methods by collecting data, conducting analysis, making discussions and conclusions. The results of this study are an analysis of various scenes from the ABC soy sauce of Suami Sejati Mau Masak which is considered to have a feminist message using Roland Barthes's theoretical approach discussing denotative, connotative and myth development.

News and Analysis of the Global Innovation Scene

News and Analysis of the Global Innovation Scene

Adaptive shift of active electroreception in weakly electric fish for troglobitic life

The adaptive-shift hypothesis for the evolution of cave-dwelling species posits that ancestor species in surface habitats had exaptations for subterranean life that were exploited when individuals invaded caves. Weakly electric Gymnotiform fishes, nocturnal South American teleost fishes, have features that appear to be likely exaptations for troglobitic life. These fishes have active electrosensory systems in which fish generate weak electric fields that are detected by specialized electroreceptors. Gymnotiform fishes use their electric fields for navigation, prey capture (scene analysis), and social communication. Although active electrosensory systems appear to be exaptations for troglobitic life, as fish use these systems to “see in the dark”, producing electric fields is energetically costly. Cave habitats, which often are low in resources, may not be able to support such high energetic demands. Eigenmannia vicentespelaea, a species of weakly electric fish that is endemic to the São Vicente II cave in central Brazil, surprisingly generates stronger electric fields than their surface relatives. The increase in strength of electric fields may result simply from differences in size between cave and surface populations, but may also be due to lack of predation pressure in the cave or increases in “sensory volumes” and acuity that improve prey localization and capture. Eigenmannia vicentespelaea exhibits the classical phenotypes of any troglobitic fish: these fish have small to nonexistent eyes and loss of pigmentation. The closest living surface relative, Eigenmannia trilineata, inhabits streams nearby and has eyes and pigmentation. The electrosensory and locomotor behavior of both species of fish were measured in their natural habitats using a grid recording system. Surface Eigenmannia exhibited dramatic circadian changes in social behavior, such as hiding under rocks during the day and foraging in groups at night, while cave Eigenmannia displayed territorial behavior with no apparent circadian modulations. The territorial behavior involved electrical and movement-based interactions that may be a form of boundary patrolling. Electrosocial behavior and scene analysis are mechanistically interlinked because both stem from active sensing tactics.

Challenges and Solutions for Autonomous Ground Robot Scene Understanding and Navigation in Unstructured Outdoor Environments: A Review

The capabilities of autonomous mobile robotic systems have been steadily improving due to recent advancements in computer science, engineering, and related disciplines such as cognitive science. In controlled environments, robots have achieved relatively high levels of autonomy. In more unstructured environments, however, the development of fully autonomous mobile robots remains challenging due to the complexity of understanding these environments. Many autonomous mobile robots use classical, learning-based or hybrid approaches for navigation. More recent learning-based methods may replace the complete navigation pipeline or selected stages of the classical approach. For effective deployment, autonomous robots must understand their external environments at a sophisticated level according to their intended applications. Therefore, in addition to robot perception, scene analysis and higher-level scene understanding (e.g., traversable/non-traversable, rough or smooth terrain, etc.) are required for autonomous robot navigation in unstructured outdoor environments. This paper provides a comprehensive review and critical analysis of these methods in the context of their applications to the problems of robot perception and scene understanding in unstructured environments and the related problems of localisation, environment mapping and path planning. State-of-the-art sensor fusion methods and multimodal scene understanding approaches are also discussed and evaluated within this context. The paper concludes with an in-depth discussion regarding the current state of the autonomous ground robot navigation challenge in unstructured outdoor environments and the most promising future research directions to overcome these challenges.

The Teaching of Doctrine in Casarosa’s Film “Luca”

Background: Films seen from the element of doctrine are very interesting to be used as positive lessons, because every action can have consequences for choices in attitude. Purpose: This study aims to observe the doctrine in the film "Luca" by Enrico Casarosa, so that it can be known the consequences must be accepted by Luca as the main character, the attitude that must be carried out in response to the doctrine. Method: This research is a literature research. The primary data source was obtained by Casarosa's film "Luca" published in 2021. The data analysis method uses content-based qualitative analysis of dialogue words, sentences, images, cuts, scenes, paragraphs, and descriptive film narratives as well as the responses of the main character "Luca" who have elements of doctrination. Data analysis techniques are carried out through three stages: data reduction, data display, and conclusion pulling/verification Results and Discussion: The results of this study show that there is a doctrine in the film "Luca" by Casarosa. Doctrine against the main character, Luca: Among them are 1) obedience, which is marked by running into hiding; he must not have contact with man; It is not allowed to talk, discuss, and even approach the surface of the sea. As a consequence, if Luca does not comply, he will be captured and killed by humans. 2) Freedom, which is characterized by humans having their own vehicle in life, does not need anyone without pressure. As a consequence, if he does not follow Alberto's character, he will not get freedom, cannot adventure in the human world, and will still feel afraid of everything. 3) Continue to learn, create and innovate, this is indicated by the existence of a telescope that can observe distant objects seen closely; You can travel in the universe and galaxies only through small tools such as telescopes. As a consequence, if he hadn't followed Giulia, he would never have had the experience of a telescope adventure. This tells us about the importance of going to school. Conclusions and Implications: The film "Luca", Casarosa inserts several doctrines to serve as lessons such as: 1) Whenever we see danger or murderers, we need to run to hide. Unless, we will be killed. 2) Whenever we have worries or fears in mind, we need to encourage them all and build our own ways to manage them as a force. Unless, we will continue to feel afraid of facing problems. 3) Dare to learn more, to experience more, to adventure more, especially to the world, universe, and galaxy, then our knowledge will increase. If we just stay and do nothing, we will only continue to be like a frog in a shell. 4) Learn more to increase knowledge, then we will know how to adventure the world and universe. If we do not develop ourselves by learning, then we will only become like people who stutter in knowledge.

Deep-Learning-Based Point Cloud Semantic Segmentation: A Survey

With the rapid development of sensor technologies and the widespread use of laser scanning equipment, point clouds, as the main data form and an important information carrier for 3D scene analysis and understanding, play an essential role in the realization of national strategic needs, such as traffic scene perception, natural resource management, and forest biomass carbon stock estimation. As an important research direction in 3D computer vision, point cloud semantic segmentation has attracted more and more researchers’ attention. In this paper, we systematically outline the main research problems and related research methods in point cloud semantic segmentation and summarize the mainstream public datasets and common performance evaluation metrics. Point cloud semantic segmentation methods are classified into rule-based methods and point-based methods according to the representation of the input data. On this basis, the core ideas of each type of segmentation method are introduced, the representative and innovative algorithms of each type of method are elaborated, and the experimental results on the datasets are compared and analyzed. Finally, some promising research directions and potential tendencies are proposed.

TCANet: three-stream coordinate attention network for RGB-D indoor semantic segmentation

Semantic segmentation plays a vital role in indoor scene analysis. Currently, its accuracy is still limited due to the complex conditions of various indoor scenes. In addition, it is difficult to complete this task solely relying on RGB images. Since depth images can provide additional 3D geometric information to RGB images, researchers chose to incorporate depth images for improving the accuracy of indoor semantic segmentation. However, it is still a challenge to effectively fuse the depth information with the RGB images. To address this issue, a three-stream coordinate attention network is proposed. The presented network reconstructs a multi-modal feature fusion module for RGB-D features, which can realize the aggregation of two modal information along the spatial and channel dimensions. Meanwhile, three convolutional neural network branches are used to construct a parallel three-stream structure, which can, respectively, process the RGB features, depth features and combined features. On one hand, the proposed network can preserve the original RGB and depth feature streams, simultaneously. On the other hand, it can also contribute to utilize and propagate the fusion feature flow better. The embedded ASPP module is used to optimize the semantic information in the proposed network, so as to aggregate the feature information of different scales and obtain more accurate features. Experimental results show that the proposed model can reach a state-of-the-art mIoU accuracy of 50.2% on the NYUDv2 dataset and on the more complex SUN-RGBD dataset.

Utilizing Drone-Based Ground-Penetrating Radar for Crime Investigations in Localizing and Identifying Clandestine Graves.

The decomposition of a body is influenced by burial conditions, making it crucial to understand the impact of different conditions for accurate grave detection. Geophysical techniques using drones have gained popularity in locating clandestine graves, offering non-invasive methods for detecting surface and subsurface irregularities. Ground-penetrating radar (GPR) is an effective technology for identifying potential grave locations without disturbance. This research aimed to prototype a drone system integrating GPR to assist in grave localization and to develop software for data management. Initial experiments compared GPR with other technologies, demonstrating its valuable applicability. It is suitable for various decomposition stages and soil types, although certain soil compositions have limitations. The research used the DJI M600 Pro drone and a drone-based GPR system enhanced by the real-time kinematic (RTK) global positioning system (GPS) for precision and autonomy. Tests with simulated graves and cadavers validated the system's performance, evaluating optimal altitude, speed, and obstacle avoidance techniques. Furthermore, global and local planning algorithms ensured efficient and obstacle-free flight paths. The results highlighted the potential of the drone-based GPR system in locating clandestine graves while minimizing disturbance, contributing to the development of effective tools for forensic investigations and crime scene analysis.

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image.

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

Improved Short-term Dense Bottleneck network for efficient scene analysis

Visual scene understanding mainly depends on pixel-wise classification obtained from a deep convolutional neural network. However, existing semantic segmentation models often face difficulties in real-time applications due to their large network architecture. Although there are real-time semantic segmentation models available, their shallow backbone can degrade the performance considerably. This paper introduces SDBNetV2, a lightweight semantic segmentation model designed to improve real-time performance without increasing computational costs. A key contribution is a novel Short-term Dense Bottleneck (SDB) module in the encoder, which provides varied field-of-views to capture different geometrical objects in a complex scene. Additionally, we propose dense feature refinement and improved semantic aggregation modules at the decoder end to enhance contextualization and object localization. We evaluate the proposed model’s performance on several indoor and outdoor datasets in structured and unstructured environments. The results show that SDBNetV2 achieves superior segmentation performance over other real-time models with less than 2 million parameters.