Object Representation Research Articles

Attention redirection transformer with semantic oriented learning for unbiased scene graph generation

Scene Graph Generation (SGG) plays an important role in scene understanding because all of the objects and relations in an image can be abstracted into a concise topological graph. Due to the complexity of visual scenes, including mutual occlusion between objects and semantic ambiguity, SGG is still a challenging task. Most of the existing models only focused on the context of a single object while contexts provided by paired objects are ignored. In this paper, we propose an Attention Redirection Transformer (ART) to extract pair-level contexts specifically, which is divided into an attention distraction stage and an attention integration stage. In this way, the attention of the model is forced to be redirected, which explores the implicit information in the background. In addition, to incorporate the semantic information of predicates, a Semantic Oriented Learning Module (SOL) is designed, which may assist in getting better textual semantics and also prompts cross-modal information fusion. At last, a self-diversity driven Dual Translation Embedding Module (DTM) is designed, which refines representations of subject and object and makes them distinct. Experimental results on the Visual Genome dataset demonstrate the effectiveness of our proposed method. Moreover, our method outperforms state-of-the-art methods on the mR@K metric. The source codes are released on Github: https://github.com/Nora-Zhang98/ART-SOL.

From CAD to representations suitable for isogeometric analysis: a complete pipeline

AbstractNext generation CAD/CAE systems shall integrate design and analysis procedures as a means to simplify workflows, boost performance, and speed up time to market. In the present paper we demonstrate a proof of concept of such a unification using subdivision surfaces and the enabling technology of isogeometric analysis. In particular, we present a complete pipeline to convert CAD models into smooth $$G^1$$ G 1 spline representations, which are suitable for isogeometric analysis. Starting from a CAD boundary representation of a mechanical object, we perform an automatic control cage extraction by means of quadrangular faces, such that its limit Catmull-Clark subdivision surface approximates accurately the input model. We then compute a basis of the $$G^1$$ G 1 spline space over the quad mesh in order to carry out least squares fitting over a point cloud, acquired by sampling the original CAD geometry. The resulting surface is a collection of Bézier patches with $$G^1$$ G 1 regularity. Finally, we use the basis functions to perform isogeometric analysis simulations of realistic PDEs on the reconstructed $$G^1$$ G 1 model. The quality of the construction is demonstrated via several numerical examples performed on a collection of CAD objects presenting various challenging realistic shapes.

Fast frequency modulation is encoded according to the listener expectations in the human subcortical auditory pathway

Abstract Expectations aid and bias our perception. For instance, expected words are easier to recognise than unexpected words, particularly in noisy environments, and incorrect expectations can make us misunderstand our conversational partner. Expectations are combined with the output from the sensory pathways to form representations of auditory objects in the cerebral cortex. Previous literature has shown that expectations propagate further down to subcortical stations during the encoding of static pure tones. However, it is unclear whether expectations also drive the subcortical encoding of subtle dynamic elements of the acoustic signal that are not represented in the tonotopic axis. Here, we tested the hypothesis that subjective expectations drive the encoding of fast frequency modulation (FM) in the human subcortical auditory pathway. We used fMRI to measure neural responses in the human auditory midbrain (inferior colliculus) and thalamus (medial geniculate body). Participants listened to sequences of FM-sweeps for which they held different expectations based on the task instructions. We found robust evidence that the responses in auditory midbrain and thalamus encode the difference between the acoustic input and the subjective expectations of the listener. The results indicate that FM-sweeps are already encoded at the level of the human auditory midbrain and that encoding is mainly driven by subjective expectations. We conclude that the subcortical auditory pathway is integrated in the cortical network of predictive processing and that expectations are used to optimise the encoding of fast dynamic elements of the acoustic signal.

Riemannian optimization of photonic quantum circuits in phase and Fock space

We propose a framework to design and optimize generic photonic quantum circuits composed of Gaussian objects (pure and mixed Gaussian states, Gaussian unitaries, Gaussian channels, Gaussian measurements) as well as non-Gaussian effects such as photon-number-resolving measurements. In this framework, we parametrize a phase space representation of Gaussian objects using elements of the symplectic group (or the unitary or orthogonal group in special cases), and then we transform it into the Fock representation using a single linear recurrence relation that computes the Fock amplitudes of any Gaussian object recursively. We also compute the gradient of the Fock amplitudes with respect to phase space parameters by differentiating through the recurrence relation. We can then use Riemannian optimization on the symplectic group to optimize MM-mode Gaussian objects, avoiding the need to commit to particular realizations in terms of fundamental gates. This allows us to “mod out” all the different gate-level implementations of the same circuit, which now can be chosen after the optimization has completed. This can be especially useful when looking to answer general questions, such as bounding the value of a property over a class of states or transformations, or when one would like to worry about hardware constraints separately from the circuit optimization step. Finally, we make our framework extendable to non-Gaussian objects that can be written as linear combinations of Gaussian ones, by explicitly computing the change in global phase when states undergo Gaussian transformations. We implemented all of these methods in the freely available open-source library MrMustard, which we use in three examples to optimize the 216-mode interferometer in Borealis, and 2- and 3-modes circuits (with Fock measurements) to produce cat states and cubic phase states.

From 3D point‐cloud data to explainable geometric deep learning: State‐of‐the‐art and future challenges

AbstractWe present an exciting journey from 3D point‐cloud data (PCD) to the state of the art in graph neural networks (GNNs) and their evolution with explainable artificial intelligence (XAI), and 3D geometric priors with the human‐in‐the‐loop. We follow a simple definition of a “digital twin,” as a high‐precision, three‐dimensional digital representation of a physical object or environment, captured, for example, by Light Detection and Ranging (LiDAR) technology. After a digression into transforming PCD into images, graphs, combinatorial complexes and hypergraphs, we explore recent developments in geometric deep learning (GDL) and provide insight into the application of these network architectures for analyzing and learning from graph‐structured data. We emphasize the importance of the explainability of these models and recognize that the ability to interpret and validate the results of complex models is a crucial aspect of their wider adoption.This article is categorized under: Technologies > Artificial Intelligence

FMRI decoding reveals independence in object representations in visual working memory

fMRI decoding reveals independence in object representations in visual working memory

Aging delays the formation of object representations

Aging delays the formation of object representations

Using imagination and the contents of memory to create new scene and object representations: A functional MRI study

Humans can use the contents of memory to construct scenarios and events that they have not encountered before, a process colloquially known as imagination. Much of our current understanding of the neural mechanisms mediating imagination is limited by paradigms that rely on participants' subjective reports of imagined content. Here, we used a novel behavioral paradigm that was designed to systematically evaluate the contents of an individual's imagination. Participants first learned the layout of four distinct rooms containing five wall segments with differing geometrical characteristics, each associated with a unique object. During functional MRI, participants were then shown two different wall segments or objects on each trial and asked to first, retrieve the associated objects or walls, respectively (retrieval phase) and then second, imagine the two objects side-by-side or combine the two wall segments (imagination phase). Importantly, the contents of each participant's imagination were interrogated by having them make a same/different judgment about the properties of the imagined objects or scenes. Using univariate and multivariate analyses, we observed widespread activity across occipito-temporal cortex for the retrieval of objects and for the imaginative creation of scenes. Interestingly, a classifier, whether trained on the imagination or retrieval data, was able to successfully differentiate the neural patterns associated with the imagination of scenes from that of objects. Our results reveal neural differences in the cued retrieval of object and scene memoranda, demonstrate that different representations underlie the creation and/or imagination of scene and object content, and highlight a novel behavioral paradigm that can be used to systematically evaluate the contents of an individual's imagination.

Enhancing surgical navigation: a robust hand-eye calibration method for the Microsoft HoloLens 2.

Optical-see-through head-mounted displays have the ability to seamlessly integrate virtual content with the real world through a transparent lens and an optical combiner. Although their potential for use in surgical settings has been explored, their clinical translation is sparse in the current literature, largely due to their limited tracking capabilities and the need for manual alignment of virtual representations of objects with their real-world counterparts. We propose a simple and robust hand-eye calibration process for the depth camera of the Microsoft HoloLens2, utilizing a tracked surgical stylus fitted with infrared reflective spheres as the calibration tool. Using a Monte Carlo simulation and a paired-fiducial registration algorithm, we show that a calibration accuracy of 1.65mm can be achieved with as little as 6 fiducial points. We also present heuristics for optimizing the accuracy of the calibration. The ability to use our calibration method in a clinical setting is validated through a user study, with users achieving a mean calibration accuracy of 1.67mm in an average time of 42s. This work enables real-time hand-eye calibration for the Microsoft HoloLens2, without any need for a manual alignment process. Using this framework, existing surgical navigation systems employing optical or electromagnetic tracking can easily be incorporated into an augmented reality environment with a high degree of accuracy.

Method for Developing the System Architecture of Existing Industrial Objects for Digital Representation Tasks

This paper presents a method for creating the system architecture of existing industrial objects based on Model-Based Systems Engineering (MBSE) principles. The method aims to form a digital representation of physical objects, which is crucial in the digital transformation of industrial enterprises. It allows for the accurate reflection of all components, processes, functions, and interrelationships within an object. The methodology includes stages of data collection, structuring, development of ontological models, and the integration of a comprehensive system architecture into the digital space. This method was tested using a small hydroelectric power plant, revealing its key advantages and disadvantages and identifying areas for further improvement. The main findings indicate a significant improvement in understanding the system architecture for scenario modeling and digital operation of the objects. Despite challenges such as the need for multiple iterations and high data requirements, the methodology demonstrates the potential for applying MBSE in the digital transformation of existing industrial objects.

The Potential of Building Information Modeling in Agricultural Operations

Building Information Modeling (BIM) has been widely adopted in the construction industry, providing a virtual representation of physical objects and systems. In the context of agriculture, BIM can be applied to monitor and manage the physical and operational characteristics of farms and agricultural facilities using BIM digital twins. A BIM digital twin is a virtual representation of a physical object or system that is based on data from BIM models and other sources. The integration of sensors is essential in creating a BIM digital twin as they provide real-time data that can be used to update and maintain the virtual representation. The use of BIM digital twins in agriculture has the potential to improve the management of agricultural facilities, reduce environmental pollution, and promote the sustainability of agricultural land. Sensors can be used in agriculture to monitor various parameters such as soil moisture, temperature, and nutrient levels. BIM digital twins can also be used to monitor and manage sources of pollution, such as the use of pesticides and fertilizers, and to develop strategies to minimize their impact on the environment. BIM digital twins provide a comprehensive view of the condition of agricultural land and the factors that influence it, such as weather patterns, industry activities, and human activities. 4D BIM is an extension of traditional 3D BIM that adds a time dimension to the model. In the context of agriculture, 4D BIM can be used to visualize and simulate various processes related to time and soil management, such as planting and harvesting cycles, and to evaluate the impact of different planting strategies on crop yields. 4D BIM can also be used to model the impact of weather patterns on crop growth and to develop contingency plans for weather- related events. The use of BIM in agriculture has the potential to transform the industry by providing a new level of insight and control over agricultural operations. BIM models can be used to simulate and visualize different scenarios for agricultural facilities, such as the use of water, fertilizer, and energy, leading to improved efficiency and reduced waste. By using BIM to optimize operations, agricultural companies can increase their productivity and profitability. The integration of BIM digital twins and 4D BIM in agriculture has the potential to revolutionize the industry by providing new tools for monitoring and managing agricultural operations. The use of BIM digital twins and 4D BIM can help reduce environmental pollution, promote sustainability, and improve the efficiency and productivity of agricultural operations. Testing is needed to fully understand the potential benefits of BIM and BIM digital twins in agriculture and to develop best practices for their implementation.

Integration of digital twins to simulate community health scenarios and evaluate the effectiveness of public health interventions led by pharmacists

Public health emergencies such as the 2019 coronavirus disease (COVID-19) have increased the focus on digital technology. A digital twin (DT) is a cybernetic representation of a physical object (e.g., a person or a community) that helps us comprehend the complexity of the real thing and makes it easier to foresee, avoid, track, and improve its actual performance. To prepare for a new age of reliable and precise public health and medical care, DT technology can propel a much-needed radical transformation of conventional electronic health and medical records, which centre on individuals and their aggregates covering populations. Hence, this paper presents the Digital Twin-based Public Health Intervention Model (DT-PHIM) to simulate community health led by pharmacists. The use of digital twins in healthcare has the potential to revolutionize workflow analysis and resource allocation, ultimately resulting in better patient care and more efficient operations. In addition, healthcare providers may hone their abilities and rehearse intricate treatments in a secure and realistic environment with the help of digital twins. There is great promise for digital twin technology to revolutionize healthcare by boosting innovation, patient safety, and overall patient outcomes. On top of that, DTs may help doctors and other medical staff, such as hospital pharmacists, make decisions by simulating how therapy would affect these patients. The experimental outcomes demonstrate that the suggested DT-PHIM model increases the patient data access ratio by 97.4%, the monitoring efficiency ratio by 96.2% and the decision-making ratio by 98.2% compared to other existing models.

Dual temporal memory network with high-order spatio-temporal graph learning for video object segmentation

Typically, Video Object Segmentation (VOS) always has the semi-supervised setting in the testing phase. The VOS aims to track and segment one or several target objects in the following frames in the sequence, only given the ground-truth segmentation mask in the initial frame. A fundamental issue in VOS is how to best utilize the temporal information to improve the accuracy. To address the aforementioned issue, we provide an end-to-end framework that simultaneously extracts long-term and short-term historical sequential information to current frame as temporal memories. The integrated temporal architecture consists of a short-term and a long-term memory modules. Specifically, the short-term memory module leverages a high-order graph-based learning framework to simulate the fine-grained spatial–temporal interactions between local regions across neighboring frames in a video, thereby maintaining the spatio-temporal visual consistency on local regions. Meanwhile, to relieve the occlusion and drift issues, the long-term memory module employs a Simplified Gated Recurrent Unit (S-GRU) to model the long evolutions in a video. Furthermore, we design a novel direction-aware attention module to complementarily augment the object representation for more robust segmentation. Our experiments on three mainstream VOS benchmarks, containing DAVIS 2017, DAVIS 2016, and Youtube-VOS, demonstrate that our proposed solution provides a fair tradeoff performance between both speed and accuracy.

Manipulating and measuring variation in deep neural network (DNN) representations of objects

We explore how DNNs can be used to develop a computational understanding of individual differences in high-level visual cognition given their ability to generate rich meaningful object representations informed by their architecture, experience, and training protocols. As a first step to quantifying individual differences in DNN representations, we systematically explored the robustness of a variety of representational similarity measures: Representational Similarity Analysis (RSA), Centered Kernel Alignment (CKA), and Projection-Weighted Canonical Correlation Analysis (PWCCA), with an eye to how these measures are used in cognitive science, cognitive neuroscience, and vision science. To manipulate object representations, we next created a large set of models varying in random initial weights and random training image order, training image frequencies, training category frequencies, and model size and architecture and measured the representational variation caused by each manipulation. We examined both small (All-CNN-C) and commonly-used large (VGG and ResNet) DNN architectures. To provide a comparison for the magnitude of representational differences, we established a baseline based on the representational variation caused by image-augmentation techniques used to train those DNNs. We found that variation in model randomization and model size never exceeded baseline. By contrast, differences in training image frequency and training category frequencies caused representational variation that exceeded baseline, with training category frequency manipulations exceeding baseline earlier in the networks. These findings provide insights into the magnitude of representational variations that can be expected with a range of manipulations and provide a springboard for further exploration of systematic model variations aimed at modeling individual differences in high-level visual cognition.

Wave-PCT: Wavelet point cloud transformer for point cloud quality assessment

Point cloud representation for real-world objects has seen a surge in interest recently, finding widespread applications in augmented reality, virtual reality, and autonomous driving. However, the process of acquiring, compressing, transmitting, and processing point cloud data often introduces unwanted distortions, highlighting the need for effective no-reference quality assessment methods for distorted point clouds. In this paper, we propose a Wavelet Point Cloud Transformer (Wave-PCT) method to assess the quality of point clouds without reference data. We start by performing patch sampling on a source point cloud, generating an extensive set of patches from the point cloud. These point cloud patches and their 2D projection images are processed using wavelets to produce multiscale local spectral features; moreover, we also extract global features of these patches using Pointnet++. Finally, a self-attention module is performed to fuse these local and global features, resulting in a distortion-sensitive quality score metric. We evaluate the Wave-PCT method on several point cloud quality assessment datasets using various metrics, and the experimental results indicate that the proposed framework exhibits outstanding and reliable performance when compared to various point cloud quality assessment methods.

The representation theory of Brauer categories II: Curried algebra

A representation of \mathfrak{gl}(V)=V \otimes V^{\ast} is a linear map \mu \colon \mathfrak{gl}(V) \otimes M \rightarrow M satisfying a certain identity. By currying, giving a linear map \mu is equivalent to giving a linear map a \colon V \otimes M \rightarrow V \otimes M , and one can translate the condition for \mu to be a representation into a condition on a . This alternate formulation does not use the dual of V and makes sense for any object V in a tensor category \mathcal{C} . We call such objects representations of the curried general linear algebra on V . The currying process can be carried out for many algebras built out of a vector space and its dual, and we examine several cases in detail. We show that many well-known combinatorial categories are equivalent to the curried forms of familiar Lie algebras in the tensor category of linear species; for example, the titular Brauer category is the curried form of the symplectic Lie algebra. This perspective puts these categories in a new light, has some technical applications, and suggests new directions to explore.

Generalisation to novel exemplars of learned shape categories based on visual and auditory spatial cues does not benefit from multisensory information.

Although the integration of information across multiple senses can enhance object representations in memory, how multisensory information affects the formation of categories is uncertain. In particular, it is unclear to what extent categories formed from multisensory information benefit object recognition over unisensory inputs. Two experiments investigated the categorisation of novel auditory and visual objects, with categories defined by spatial similarity, and tested generalisation to novel exemplars. Participants learned to categorise exemplars based on visual-only (geometric shape), auditory-only (spatially defined soundscape) or audio-visual spatial cues. Categorisation to learned as well as novel exemplars was then tested under the same sensory learning conditions. For all learning modalities, categorisation generalised to novel exemplars. However, there was no evidence of enhanced categorisation performance for learned multisensory exemplars. At best, bimodal performance approximated that of the most accurate unimodal condition, although this was observed only for a subset of exemplars within a category. These findings provide insight into the perceptual processes involved in the formation of categories and have relevance for understanding the sensory nature of object representations underpinning these categories.

Two "What" Networks in the Human Brain.

Ungerleider and Mishkin, in their influential work that relied on detailed anatomical and ablation studies, suggested that visual information is processed along two distinct pathways: the dorsal "where" pathway, primarily responsible for spatial vision, and the ventral "what" pathway, dedicated to object vision. This strict division of labor has faced challenges in light of compelling evidence revealing robust shape and object selectivity within the putative "where" pathway. This article reviews evidence that supports the presence of shape selectivity in the dorsal pathway. A comparative examination of dorsal and ventral object representations in terms of invariance, task dependency, and representational content reveals similarities and differences between the two pathways. Both exhibit some level of tolerance to image transformations and are influenced by tasks, but responses in the dorsal pathway show weaker tolerance and stronger task modulations than those in the ventral pathway. Furthermore, an examination of their representational content highlights a divergence between the responses in the two pathways, suggesting that they are sensitive to distinct features of objects. Collectively, these findings suggest that two networks exist in the human brain for processing object shapes, one in the dorsal and another in the ventral visual cortex. These studies lay the foundation for future research aimed at revealing the precise roles the two "what" networks play in our ability to understand and interact with objects.

Introducing an indoor object classification dataset including sparse point clouds from mmWave radar

This document introduces the RadIOCD, which is a dataset that contains sparse point cloud representations of indoor objects, collected by subjects wearing a commercial off-the-shelf mmWave radar. In particular, RadIOCD includes the recordings of 10 volunteers moving towards 5 different objects (i.e., backpack, chair, desk, human, and wall), placed in 3 different environments. RadIOCD includes sparse 3D point cloud data, together with their doppler velocity and intensity provided by the mmWave radar. A total of 5,776 files are available, with each one having an approximate duration of 8s. The scope of RadIOCD is the availability of data for the recognition of objects solely recorded by the mmWave radar, to be used in applications were the vision-based classification is cumbersome though critical (e.g., in search and rescue operation where there is smoke inside a building). Furthermore, we showcase that this dataset after being segmented into 76,821 samples contains enough data to apply Machine Learning-based techniques, ensuring that they could generalize in different environments and “unseen“ subjects.

The Bastard Child of Historicism? Realisms Roots in History Painting

Summary In her classic 1974 study Realism, art historian Linda Nochlin claims that by the mid-nineteenth century, realism and realism esthetics so had permeated the art world that it also had affected the academic history painters. But what if this is wrong, if it wasn’t the history painters that were affected by the realists, but rather the other way around? In this article, I will make the case that Realism, the esthetical movement of the mid-nineteenth century, had its roots in the historicism of the early nineteenth century and that the realist claims to give a truthful, objective and impartial representation of the real world, based on meticulous observation, were already present in discussions of history and how to represent history in the arts from the years around 1800. Secondly, the study is also a contribution to the research concerning how the idea and concept of history changed in the early nineteenth century.