Intermediate Representation Level Research Articles

Representation of Natural Contours by a Neural Population in Monkey V4.

The cortical visual area, V4, has been considered to code contours that contribute to the intermediate-level representation of objects. The neural responses to the complex contour features intrinsic to natural contours are expected to clarify the essence of the representation. To approach the cortical coding of natural contours, we investigated the simultaneous coding of multiple contour features in monkey (Macaca fuscata) V4 neurons and their population-level representation. A substantial number of neurons showed significant tuning for two or more features such as curvature and closure, indicating that a substantial number of V4 neurons simultaneously code multiple contour features. A large portion of the neurons responded vigorously to acutely curved contours that surrounded the center of classical receptive field, suggesting that V4 neurons tend to code prominent features of object contours. The analysis of mutual information (MI) between the neural responses and each contour feature showed that most neurons exhibited similar magnitudes for each type of MI, indicating that many neurons showing the responses depended on multiple contour features. We next examined the population-level representation by using multidimensional scaling analysis. The neural preferences to the multiple contour features and that to natural stimuli compared with silhouette stimuli increased along with the primary and secondary axes, respectively, indicating the contribution of the multiple contour features and surface textures in the population responses. Our analyses suggested that V4 neurons simultaneously code multiple contour features in natural images and represent contour and surface properties in population.

Dimensionality of the intermediate-level representation of shape and texture in monkey V4

The visual area V4 has been considered to play a crucial role in the intermediate representation of objects, where low-level image features are transformed into object-level representations. We estimated the intrinsic dimensionality in V4 for the representation of local patches generated from natural scenes. The dimensionality was approximately 40, which is approximately half of that reported in IT for the representation of whole natural objects. The analyses of the estimated dimensionality suggest both common and independent representations that code contour shapes and/or surfaces with textures, implying a relatively complex and mixed representation in the intermediate-level area.

Memory-Aware Functional IR for Higher-Level Synthesis of Accelerators

Specialized accelerators deliver orders of a magnitude of higher performance than general-purpose processors. The ever-changing nature of modern workloads is pushing the adoption of Field Programmable Gate Arrays (FPGAs) as the substrate of choice. However, FPGAs are hard to program directly using Hardware Description Languages (HDLs). Even modern high-level HDLs, e.g., Spatial and Chisel, still require hardware expertise. This article adopts functional programming concepts to provide a hardware-agnostic higher-level programming abstraction. During synthesis, these abstractions are mechanically lowered into a functional Intermediate Representation (IR) that defines a specific hardware design point. This novel IR expresses different forms of parallelism and standard memory features such as asynchronous off-chip memories or synchronous on-chip buffers. Exposing such features at the IR level is essential for achieving high performance. The viability of this approach is demonstrated on two stencil computations and by exploring the optimization space of matrix-matrix multiplication. Starting from a high-level representation for these algorithms, our compiler produces low-level VHSIC Hardware Description Language (VHDL) code automatically. Several design points are evaluated on an Intel Arria 10 FPGA, demonstrating the ability of the IR to exploit different hardware features. This article also shows that the designs produced are competitive with highly tuned OpenCL implementations and outperform hardware-agnostic OpenCL code.

CN-waterfall: a deep convolutional neural network for multimodal physiological affect detection

Affective computing solutions, in the literature, mainly rely on machine learning methods designed to accurately detect human affective states. Nevertheless, many of the proposed methods are based on handcrafted features, requiring sufficient expert knowledge in the realm of signal processing. With the advent of deep learning methods, attention has turned toward reduced feature engineering and more end-to-end machine learning. However, most of the proposed models rely on late fusion in a multimodal context. Meanwhile, addressing interrelations between modalities for intermediate-level data representation has been largely neglected. In this paper, we propose a novel deep convolutional neural network, called CN-Waterfall, consisting of two modules: Base and General. While the Base module focuses on the low-level representation of data from each single modality, the General module provides further information, indicating relations between modalities in the intermediate- and high-level data representations. The latter module has been designed based on theoretically grounded concepts in the Explainable AI (XAI) domain, consisting of four different fusions. These fusions are mainly tailored to correlation- and non-correlation-based modalities. To validate our model, we conduct an exhaustive experiment on WESAD and MAHNOB-HCI, two publicly and academically available datasets in the context of multimodal affective computing. We demonstrate that our proposed model significantly improves the performance of physiological-based multimodal affect detection.

Postmortem accurate IR-level state recovery for deployed concurrent programs

Debugging failures of deployed concurrent software is important for quality assurance. However, such failures are difficult to debug because their behavior is non-deterministic and limited information can be obtained with conventional means. Reverse debuggers such as REPT [11] assists with debugging by recovering data values before the failure. This is achieved by using a hardware-tracer to log control-flow information, then using the information and a conventional coredump to recover data values via reverse-execution at machine-level. REPT's algorithm for data value recovery is reliable and fast. But the implementation cost is high because of its dependence on architecture. Applying REPT to more abstract IR (Intermediate Representation) level instructions to counter this yielded limited results with low accuracy compared to the original x86_64 implementation. The main reason for this is that the stack layout is abstracted at IR-level. In this paper, we present STRAB (State Recovery at Abstract-level), a collection of our proposed methods to solve these problems. STRAB works in two phases. First, the data values in the coredump are lifted from machine-level to IR-level using rich debug information (DWARF3) and a novel technique we call mid-recovery lifting, the latter helping to recover more heap data values at IR-level. Second, our novel hybrid memory location resolution reduces the accuracy loss due to the abstracted stack layout at IR-level. Experimental results on a variety of real-world concurrent programs show that STRAB has significantly higher accuracy compared to REPT at IR-level (+40% on average) with only minor slowdowns (x2.7 on average), while also achieving architecture-independence.

Structural Knowledge Distillation for Efficient Skeleton-Based Action Recognition.

Skeleton data have been extensively used for action recognition since they can robustly accommodate dynamic circumstances and complex backgrounds. To guarantee the action-recognition performance, we prefer to use advanced and time-consuming algorithms to get more accurate and complete skeletons from the scene. However, this may not be acceptable in time- and resource-stringent applications. In this paper, we explore the feasibility of using low-quality skeletons, which can be quickly and easily estimated from the scene, for action recognition. While the use of low-quality skeletons will surely lead to degraded action-recognition accuracy, in this paper we propose a structural knowledge distillation scheme to minimize this accuracy degradations and improve recognition model's robustness to uncontrollable skeleton corruptions. More specifically, a teacher which observes high-quality skeletons obtained from a scene is used to help train a student which only sees low-quality skeletons generated from the same scene. At inference time, only the student network is deployed for processing low-quality skeletons. In the proposed network, a graph matching loss is proposed to distill the graph structural knowledge at an intermediate representation level. We also propose a new gradient revision strategy to seek a balance between mimicking the teacher model and directly improving the student model's accuracy. Experiments are conducted on Kenetics400, NTU RGB+D and Penn action recognition datasets and the comparison results demonstrate the effectiveness of our scheme.

Discriminative neural network pruning in a multiclass environment: A case study in spoken emotion recognition

Deep learning has become one of the most widely accepted paradigms regarding machine learning. It focuses on the use of hierarchical data models and builds upon the notion that in order to learn about high level data representations, a better understanding of intermediate level representation is needed. Restricted Boltzmann Machines and deep belief networks are two main types of deep learning algorithms commonly used in a wide array of classification and pattern recognition tasks. Examples of these tasks are natural language recognition, neuroimaging studies, forecasting time series, parametric voice synthesis, and speech emotion recognition among others. Recent machine learning studies suggest that deep learning networks can help map feature problems into a more advantageous position, hence improving the classification process. However, selecting a suitable Deep learning architecture in response to a specific problem can be difficult. In this study, we intend to investigate whether discriminative measures, such as Anova, Pearsonâs Correlation, Fisher score, Gain ratio, ReliefF, OneR among others, could offer pointers to identify useful neural nods in a Deep learning network. This is due to the fact that normally not all hidden neurons provide insightful information for a classification task. Our approach consists in using some of these discriminative measures to rank the hidden neurons based on their output values, and then prune them in accordance to their position within said ranking. Our results indicate that this approach is also helpful in multiclass classification problems and the pruning process seems to have a positive effect in diminishing the resulting error rate.

A Representation of Rhythmic Motions

Skill is an action requiring intelligence. It has been debated whether motions require representations because they are thought to describe procedures. The dichotomy between thought and execution has puzzled us because we are unsure how thinking process intervenes with execution and how execution is understood by thinking process. We have in other words no logical theory of executions. We propose an intermediate level of representation between thought and execution. Hinted by an illustration by a professional musician, we propose to represent the skill knowledge as a phase space. The shape conveys necessary items of information to dance and play samba with different types of instruments thanks to the abstract representation of motions. We show that the shape is established by abstracting over physical particulars such as velocity, direction, position, etc. The abstract form of representation bridges the gap between thought and execution. It also allows masters to transfer his knowledge to pupils. We hypothesise that more layers would exist between thought and execution, enabling the agent to effectively plan and control his motions.

Zero-Shot Classification Based on Multitask Mixed Attribute Relations and Attribute-Specific Features

Zero-shot classification is a hot topic in computer vision and pattern recognition. Most zero-shot classification methods are based on the intermediate level representation of attributes to achieve knowledge transfer from the training classes to the unseen test classes. Recently, multitask learning (MTL) has been shown as one of state-of-the-art approaches for attribute learning and zero-shot classification. Aiming at the attribute relation learning, features shared by attributes learning and attribute heterogeneity, we propose a zero-shot classification based on multitask mixed attribute relations and attribute-specific features. First, considering the relationship between attribute–attribute and attribute–features, a second-order attribute relation and attribute-specific features learning model is constructed from training samples based on MTL. Second, second-order attribute relation is extended to high-order attribute relation and multiple attribute classifiers are learned. Finally, zero-shot classification is completed based on the maximum posterior probability. Experimental results on AWA and PubFig data sets show that the proposed method can yield more accurate attribute prediction and zero-shot classification compared with several multitask attribute learning methods.

Representations and Optimizations for Embedded Parallel Dataflow Languages

Parallel dataflow engines such as Apache Hadoop, Apache Spark, and Apache Flink are an established alternative to relational databases for modern data analysis applications. A characteristic of these systems is a scalable programming model based on distributed collections and parallel transformations expressed by means of second-order functions such as map and reduce. Notable examples are Flink’s DataSet and Spark’s RDD programming abstractions. These programming models are realized as EDSLs—domain specific languages embedded in a general-purpose host language such as Java, Scala, or Python. This approach has several advantages over traditional external DSLs such as SQL or XQuery. First, syntactic constructs from the host language (e.g., anonymous functions syntax, value definitions, and fluent syntax via method chaining) can be reused in the EDSL. This eases the learning curve for developers already familiar with the host language. Second, it allows for seamless integration of library methods written in the host language via the function parameters passed to the parallel dataflow operators. This reduces the effort for developing analytics dataflows that go beyond pure SQL and require domain-specific logic. At the same time, however, state-of-the-art parallel dataflow EDSLs exhibit a number of shortcomings. First, one of the main advantages of an external DSL such as SQL—the high-level, declarative Select-From-Where syntax—is either lost completely or mimicked in a non-standard way. Second, execution aspects such as caching, join order, and partial aggregation have to be decided by the programmer. Optimizing them automatically is very difficult due to the limited program context available in the intermediate representation of the DSL. In this article, we argue that the limitations listed above are a side effect of the adopted type-based embedding approach. As a solution, we propose an alternative EDSL design based on quotations. We present a DSL embedded in Scala and discuss its compiler pipeline, intermediate representation, and some of the enabled optimizations. We promote the algebraic type of bags in union representation as a model for distributed collections and its associated structural recursion scheme and monad as a model for parallel collection processing. At the source code level, Scala’s comprehension syntax over a bag monad can be used to encode Select-From-Where expressions in a standard way. At the intermediate representation level, maintaining comprehensions as a first-class citizen can be used to simplify the design and implementation of holistic dataflow optimizations that accommodate for nesting and control-flow. The proposed DSL design therefore reconciles the benefits of embedded parallel dataflow DSLs with the declarativity and optimization potential of external DSLs like SQL.

Semantic convex matrix factorisation for cross‐media retrieval

When utilising matrix factorisation to extract latent features for cross-media retrieval, semantic information may be lost in the process of factorisation. In addition, many presented approaches directly mapped different modalities into an isomorphic semantic space to conduct the similarity measurement of different modalities, which also resulted in the loss of crucial information. To address these problems, a semantic convex matrix factorisation subspace learning approach is proposed for cross-media retrieval between image and text. The proposed method can extract an intermediate-level feature representation for the high dimensional image modality in order to weaken the loss of information, in the meantime, learn a semantic feature representation with semantic information for the lower dimension text modality to strengthen the discriminated capability. After that, the intermediate-level feature representation of image is mapped into a latent semantic space by a projection matrix. Then the similarity of different modalities can be estimated in terms of uniform dimensional latent feature representations. Experimental results on three benchmark datasets demonstrate the superiority of the proposed approach over several state-of-the-art approaches.

Dependable Deep Computation Model for Feature Learning on Big Data in Cyber-Physical Systems

With the ongoing development of sensor devices and network techniques, big data are being generated from the cyber-physical systems. Because of sensor equipment occasional failure and network transmission unreliability, a large number of low-quality data, such as noisy data and incomplete data, is collected from the cyber-physical systems. Low-quality data pose a remarkable challenge on deep learning models for big data feature learning. As a novel deep learning model, the deep computation model achieves superior performance for big data feature learning. However, it is difficult for the deep computation model to learn dependable features for low-quality data, since it uses the nonlinear function as the encoder. In this article, a dependable deep computation model is proposed for feature learning on low-quality big data in cyber-physical systems. Specially, a regularity is added into the objective function of the deep computation model to obtain reliable features in the intermediate-level representation space. Furthermore, a learning algorithm based on the back-propagation strategy is devised to train the parameters of the proposed model. Finally, experiments are conducted on three representative datasets and a real dataset to evaluate the effectiveness of the dependable deep computation model for low-quality big data feature learning. Results show that the proposed model achieves a remarkable result for the tasks of classification, restoration, and prediction, proving the potential of this work for practical applications in cyber-physical systems.

Sharing teacher knowledge at scale: teacher inquiry, learning design and the representation of teachers’ practice

Sharing teachers’ professional knowledge remains challenging. Teachers’ development often remains ad hoc or local, and attempts to scale this up have proved problematic. To address this, research in areas such as ‘learning design’ has explored the use of formal representations of practice. This proposes that educational practice can be improved by documenting, sharing and building on what teachers already do. Whilst this has led to some successes, it has not resulted in the widespread transformation of practice. This paper reviews the literature about sharing teacher knowledge. The challenges of scaling up development are then considered in relation to two theories that help explain the challenges: Communities of Practice and Sociomateriality. This analysis is illustrated through case studies in Norway and the UK. These show that teachers already create and share artefacts that represent their pedagogic knowledge. However, they found formal representations, such as learning designs, difficult to work with. The paper concludes that scaling up teacher development using abstract formalisms is unlikely to succeed. Instead, teachers value stories and the materials they already create in their day-to-day practice. It is this intermediate level of representation, between direct experience and formal abstraction, that offers the most promise for sharing practice.

Energy Transparency for Deeply Embedded Programs

Energy transparency is a concept that makes a program’s energy consumption visible, from hardware up to software, through the different system layers. Such transparency can enable energy optimizations at each layer and between layers, as well as help both programmers and operating systems make energy-aware decisions. In this article, we focus on deeply embedded devices, typically used for Internet of Things (IoT) applications, and demonstrate how to enable energy transparency through existing static resource analysis (SRA) techniques and a new target-agnostic profiling technique, without hardware energy measurements. Our novel mapping technique enables software energy consumption estimations at a higher level than the Instruction Set Architecture (ISA), namely the LLVM intermediate representation (IR) level, and therefore introduces energy transparency directly to the LLVM optimizer. We apply our energy estimation techniques to a comprehensive set of benchmarks, including single- and multithreaded embedded programs from two commonly used concurrency patterns: task farms and pipelines. Using SRA, our LLVM IR results demonstrate a high accuracy with a deviation in the range of 1% from the ISA SRA. Our profiling technique captures the actual energy consumption at the LLVM IR level with an average error of 3%.

The Two-Body Inversion Effect

How does one perceive groups of people? It is known that functionally interacting objects (e.g., a glass and a pitcher tilted as if pouring water into it) are perceptually grouped. Here, we showed that processing of multiple human bodies is also influenced by their relative positioning. In a series of categorization experiments, bodies facing each other (seemingly interacting) were recognized more accurately than bodies facing away from each other (noninteracting). Moreover, recognition of facing body dyads (but not nonfacing body dyads) was strongly impaired when those stimuli were inverted, similar to what has been found for individual bodies. This inversion effect demonstrates sensitivity of the visual system to facing body dyads in their common upright configuration and might imply recruitment of configural processing (i.e., processing of the overall body configuration without prior part-by-part analysis). These findings suggest that facing dyads are represented as one structured unit, which may be the intermediate level of representation between multiple-object (body) perception and representation of social actions.

Disentangling the Representation of Identity from Head View Along the Human Face Processing Pathway

Neural models of a distributed system for face perception implicate a network of regions in the ventral visual stream for recognition of identity. Here, we report a functional magnetic resonance imaging (fMRI) neural decoding study in humans that shows that this pathway culminates in the right inferior frontal cortex face area (rIFFA) with a representation of individual identities that has been disentangled from variable visual features in different images of the same person. At earlier stages in the pathway, processing begins in early visual cortex and the occipital face area with representations of head view that are invariant across identities, and proceeds to an intermediate level of representation in the fusiform face area in which identity is emerging but still entangled with head view. Three-dimensional, view-invariant representation of identities in the rIFFA may be the critical link to the extended system for face perception, affording activation of person knowledge and emotional responses to familiar faces.

Sparse coding generates curvature selectivity in V4 neurons.

The cortical area V4 produces a representation of curvature as the intermediate-level representation of an object's shape. We investigated whether sparse coding is the principle driving the generation of the spatial properties of the receptive field in V4 that exhibit curvature selectivity. To investigate the role of sparseness in the construction of curvature representations, we applied component analysis with a sparseness constraint to the activity of model V2 neurons that were responding to shapes derived from natural images. Our simulation results showed that single basis functions with medium degrees of sparseness (0.7-0.8) produced curvature selectivity, and their population activity produced acute curvature bias. The results support the hypothesis that sparseness plays an essential role in the construction of curvature selectivity in V4.

BioC-compatible full-text passage detection for protein-protein interactions using extended dependency graph.

There has been a large growth in the number of biomedical publications that report experimental results. Many of these results concern detection of protein–protein interactions (PPI). In BioCreative V, we participated in the BioC task and developed a PPI system to detect text passages with PPIs in the full-text articles. By adopting the BioC format, the output of the system can be seamlessly added to the biocuration pipeline with little effort required for the system integration. A distinctive feature of our PPI system is that it utilizes extended dependency graph, an intermediate level of representation that attempts to abstract away syntactic variations in text. As a result, we are able to use only a limited set of rules to extract PPI pairs in the sentences, and additional rules to detect additional passages for PPI pairs. For evaluation, we used the 95 articles that were provided for the BioC annotation task. We retrieved the unique PPIs from the BioGRID database for these articles and show that our system achieves a recall of 83.5%. In order to evaluate the detection of passages with PPIs, we further annotated and Results sections of 20 documents from the dataset and show that an f-value of 80.5% was obtained. To evaluate the generalizability of the system, we also conducted experiments on AIMed, a well-known PPI corpus. We achieved an f-value of 76.1% for sentence detection and an f-value of 64.7% for unique PPI detection.Database URL: http://proteininformationresource.org/iprolink/corpora

Concepts, contents, and consciousness.

In his paper ‘Are we ever aware of concepts? A critical question for the Global Neuronal Workspace, Integrated Information, and Attended Intermediate-Level Representation theories of consciousness’ (2015, this journal), Kemmerer defends a conservative account of consciousness, according to which concepts and thoughts do not characterize the contents of consciousness, and then uses that account to argue against both the Global Neuronal Workspace theory of consciousness and Integrated Information Theory of Consciousness, and as a point in favour of Prinz’s Attended Intermediate-level Representations theory. We argue that there are a number of respects in which the contrast between conservative and liberal conceptions of the admissible contents of consciousness is more complex than Kemmerer’s discussion suggests. We then consider Kemmerer’s case for conservatism, arguing that it lumbers liberals with commitments that they need not – and in our view should not – endorse. We also argue that Kemmerer’s attempt to use his case for conservatism against the Global Neuronal Workspace and Integrated Information theories of consciousness on the one hand, and as a point in favour of Prinz’s Attended Intermediate Representations theory on the other hand, is problematic. Finally, we consider Kemmerer’s overall strategy of using an account of the admissible contents of consciousness to evaluate theories of consciousness, and suggest that here too there are complications that Kemmerer’s discussion overlooks.

The granularity of grasping: Comment on “Grasping synergies: A motor-control approach to the mirror neuron mechanism” by A. D'Ausilio et al.

The idea that mirror neuron systems in the human and the macaque monkey could provide a link between perceiving an action and performing it has spurred intense research [1,2]. Hundreds of papers now examine if this link exists and what it might contribute to human behaviour. The review article from D’Ausilio et al. [3] highlights how relatively few papers have considered the granularity of coding with mirror neuron systems, and even fewer have directly tested different possibilities. Granularity refers to the critical question of what actually is encoded within the mirror system – are neurons selective for low level kinematic features such as joint angle, or for postural synergies, or for action goals? Focusing on studies of single neurons in macaques and on studies measuring the excitability of primary motor cortex with TMS, the review suggests that it is very hard to distinguish low-level kinematic from goal representations. Furthermore, these two levels are often highly correlated in real-life contexts – the kinematics needed to grasp an apple are defined by the shape of the goal (an apple tends to be a large sphere) and these kinematics differ for other possible goals (a pencil which is a narrow cylinder). In some cases, kinematics may be enough to define a goal [4]. The review suggests that it is therefore arbitrary to distinguish these levels, and that a synergy level might be a better way to understand the mirror system. Synergies are a form of coding based on commonly used hand-shapes or hand postures, which take into account the fact that some joint angles are more likely to co-occur than others. Evidence that different grasp shapes are represented separately in premotor cortex has been found [5]. These could provide an intermediate level of representation between muscle activity and goals. The review proposes that a synergy level of granularity provides the best way to consider both the motor system and the role of the mirror system in understanding actions. This is a useful proposal, but has some limitations. The proposed synergies seem much closer to a kinematic representation than to goals, and do not necessarily resolve the problem of if/how the mirror neuron system could contribute to understanding goals. Just because it is not easy to separate cognitive concepts such as kinematics and goals, does not mean it should not be done. A small number of carefully designed fMRI experiments have been able to show distinct patterns of brain activity for goals and kinematics. For example, grasping a tennis ball and grasping an apple both involve the same hand kinematics and the same hand synergies, but quite different goals. Anterior intraparietal sulcus can distinguish these actions [6] and shows cross-modal sensitivity to other action goals too [7].