Visual Object Localization Research Articles

Optimizing pick-place operations: Leveraging k-means for visual object localization and decision-making in collaborative robots

This article presents an approach to object localization algorithms for pick-place operations in collaborative robots by utilizing conventional color segmentation in computer vision and k-means clustering. Adding the k-means clustering algorithm complements the color segmentation by distinguishing and grouping the sections of similar pixels; hence, object localization is more accurate. The order of pick-place operations of each cluster acquired from the proposed algorithm is prioritized based on norm. Integrating the proposed framework provides a well-structured depiction of the localized objects, which is fundamental for successful pick-place operations. The TCP/IP communication framework via socket communication is established to facilitate data transmission between the robot and the host computer. The objective is to ensure that the robot's end effector performs as directed by the host computer by obtaining information on the pick-and-place operation, including the localized coordinates, dimensions, the order of operations, and the pose of the objects of interest to the robot. In this experiment, a cobot arm is employed to autonomously pick and place objects with different shapes and colors in a workspace filled with diverse objects, requiring the robot to choose the closest objects to operate based on the data from the host computer. Our results demonstrate the effectiveness of this integration, showcasing the enhanced adaptability and efficiency of pick-place operations in collaborative robots. This study indicates 98% accuracy in pick-and-place operations with an average latency of 0.52 ± 0.1 s, indicating an improvement compared to the traditional algorithm without k-means clustering, which achieves an accuracy of 88%. Additional studies reveal that when incorporating pose estimation into the pick-place operations, the proposed algorithm's accuracy is 94%. The demonstration highlights the potential of leveraging machine learning algorithms and computer vision from the camera to perform flexible pick-place operations via socket communication.

Learning Visual Localization of a Quadrotor Using Its Noise as Self-Supervision

We introduce an approach to train neural network models for visual object localization using a small training set, labeled with ground truth object positions and a large unlabeled one. We assume that the object to be localized emits sound, which is perceived by a microphone rigidly affixed to the camera. This information is used as the target of a cross-modal pretext task: predicting sound features from camera frames. By solving the pretext task, the model draws self-supervision from visual and audio data. The approach is well suited to robot learning: we instantiate it to localize a small quadrotor from 128 × 80 pixel images acquired by a ground robot. Experiments on a separate testing set show that introducing the auxiliary pretext task yields large performance improvements: the Mean Absolute Error (MAE) of the estimated image coordinates of the target is reduced from 7 to 4 pixels; the MAE of the estimated distance is reduced from 28 cm to 14 cm. A model that has access to labels for the entire training set yields an MAE of 2 pixels and 11 cm, respectively.

Weakly Supervised Representation Learning for Audio-Visual Scene Analysis

Audio-visual (AV) representation learning is an important task from the perspective of designing machines with the ability to understand complex events. To this end, we propose a novel multimodal framework that instantiates multiple instance learning. Specifically, we develop methods that identify events and localize corresponding AV cues in unconstrained videos. Importantly, this is done using weak labels where only video-level event labels are known without any information about their location in time. We show that the learnt representations are useful for performing several tasks such as event/object classification, audio event detection, audio source separation and visual object localization. An important feature of our method is its capacity to learn from unsynchronized audio-visual events. We also demonstrate our framework's ability to separate out the audio source of interest through a novel use of nonnegative matrix factorization. State-of-the-art classification results, with a F1-score of 65.0, are achieved on DCASE 2017 smart cars challenge data with promising generalization to diverse object types such as musical instruments. Visualizations of localized visual regions and audio segments substantiate our system's efficacy, especially when dealing with noisy situations where modality-specific cues appear asynchronously.

Object manipulation with a variable-stiffness robotic mechanism using deep neural networks for visual semantics and load estimation

In recent years, the computer vision applications in the robotics have been improved to approach human-like visual perception and scene/context understanding. Following this aspiration, in this study, we explored the possibility of better object manipulation performance by connecting the visual recognition of objects to their physical attributes, such as weight and center of gravity (CoG). To develop and test this idea, an object manipulation platform is built comprising a robotic arm, a depth camera fixed at the top center of the workspace, embedded encoders in the robotic arm mechanism, and microcontrollers for position and force control. Since both the visual recognition and force estimation algorithms use deep learning principles, the test set-up was named as Deep-Table. The objects in the manipulation tests are selected from everyday life and are common to be seen on modern office desktops. The visual object localization and recognition processes are performed from two distinct branches by deep convolutional neural network architectures. We present five of the possible cases, having different levels of information availability on the object weight and CoG in the experiments. The results confirm that using our algorithm, the robotic arm can move different types of objects successfully varying from several grams (empty bottle) to around 250 g (ceramic cup) without failure or tipping. The proposed method also shows that connecting the object recognition with load estimation and contact point further improves the performance characterized by a smoother motion.

Logistic Regression Region Weighting for Weakly Supervised Object Localization

In this paper, we address the problem of weakly supervised object localization using region weighting. For a weakly labelled image/video, the inside regions have different relevance to its semantic label. We first over-segment an image/video to get super-pixel/voxel regions, and assign each region with a latent weight to represent its support to the semantic label, then regress the weights to right values by optimizing the classification according to the weak labels. We adopt logistic regression as our base model due to its good performance in multiple-instance setting. The latent region weights are incorporated into the objective function as an interpretation of region combination at feature-level. The weights and the model parameters are optimized in an alternate manner. With the updates of the weights, the model is trained on the semantic regions independently of the background, therefore the learned model is capable of distinguishing object and non-object regions, and generating irregular-shape object localization. The method overcomes the limitations of applying multiple-instance learning to visual object localization. Experimental results on three datasets validates the effectiveness of the proposed method.

A Novel User Experience Study of Parallax Scrolling using Eye Tracking and User Experience Questionnaire

Parallax scrolling technique is being devoted as a unique and an innovative trend in the web design. Parallax scrolling provides 3D perception on a web page. Previous works observed user experience issues of parallax scrolling merely based on subjective questionnaires. Their findings leave a research question whether the results are valid as a written questionnaire may be perceived differently by participants. Additionally, bias and ambiguity in the questionnaire can affect the research results significantly. To solve this research problem, we present a novel user experience study of parallax scrolling in storytelling and online shop website using eye tracking and User Experience Questionnaire (UEQ). Forty ( N =40) participant joined the experiment on a voluntary basis. Each participant only interacted with one out of two websites (storytelling or online shop) and only one effect (with or without parallax scrolling). We found that parallax scrolling affected UEQ score of Attractiveness of the storytelling website ( p < 0.05). Our findings suggest that parallax scrolling improves user engagement in storytelling website. We also observed that the participants spent time almost two times faster to find an object of interest in an online shop with parallax scrolling compared with the similar task in an online shop without parallax scrolling ( p < 0.05). We thus argue that parallax scrolling is useful during interacting with particular websites that require visual object localization. In future, web designers should consider the appropriate usage of parallax scrolling to optimize user experience while avoiding additional distraction caused by this technique.

Visual Space Constructed by Saccade Motor Maps.

How visual space is represented in the brain is an open question in neuroscience. Embodiment theories propose that spatial perception is structured by neural motor maps. Especially, maps which code the targets for saccadic eye movements contain a precise representation of external space. In this review article, we examine how modifications in saccade maps are accompanied by changes in visual space perception. Saccade adaptation, a method which systematically modifies saccade amplitudes, alters the localization of visual objects in space. We illustrate how information about saccade amplitudes is transferred from the cerebellum (CB) to the frontal eye field (FEF). We argue that changes in visual localization after adaptation of saccade maps provide evidence for a shared representation of visual and motor space.

Visual Mislocalization of Moving Objects in an Audiovisual Event.

The present study investigated the influence of an auditory tone on the localization of visual objects in the stream/bounce display (SBD). In this display, two identical visual objects move toward each other, overlap, and then return to their original positions. These objects can be perceived as either streaming through or bouncing off each other. In this study, the closest distance between object centers on opposing trajectories and tone presentation timing (none, 0 ms, ± 90 ms, and ± 390 ms relative to the instant for the closest distance) were manipulated. Observers were asked to judge whether the two objects overlapped with each other and whether the objects appeared to stream through, bounce off each other, or reverse their direction of motion. A tone presented at or around the instant of the objects’ closest distance biased judgments toward “non-overlapping,” and observers overestimated the physical distance between objects. A similar bias toward direction change judgments (bounce and reverse, not stream judgments) was also observed, which was always stronger than the non-overlapping bias. Thus, these two types of judgments were not always identical. Moreover, another experiment showed that it was unlikely that this observed mislocalization could be explained by other previously known mislocalization phenomena (i.e., representational momentum, the Fröhlich effect, and a turn-point shift). These findings indicate a new example of crossmodal mislocalization, which can be obtained without temporal offsets between audiovisual stimuli. The mislocalization effect is also specific to a more complex stimulus configuration of objects on opposing trajectories, with a tone that is presented simultaneously. The present study promotes an understanding of relatively complex audiovisual interactions beyond simple one-to-one audiovisual stimuli used in previous studies.

Misperceiving space following shifts of attention: Determining the locus of the attentional repulsion effect

The Attentional Repulsion Effect (ARE) is a spatial consequence of allocating attention to peripheral locations. Specifically, a shift in attention to the periphery produces an error in the localization of visual objects in the opposite direction of the shift. Suzuki and Cavanagh proposed three possible mechanisms to account for the neural basis of the repulsion effect; surround suppression, RF shrinking, and RF recruitment. The purpose of the present experiment was to begin to investigate the ARE mechanism by determining its locus. This was achieved by assessing both monocular and binocular vision, and then by examining AREs in a condition that assesses interocular transfer, where the cue is only presented to one eye and the target is only presented to the other eye. Importantly, this latter test allowed for a direct verification of the ARE’s locus. Given that the magnitude of the repulsion effect was nearly identical in the two vision conditions, and that the effect disappeared in the interocular condition, the current findings indicate that the ARE occurs early in visual processing, prior to or in the input layers of V1 and before binocular vision dominates.

Product-related sounds speed visual search

Searching for a particular product in a supermarket can be a challenging business. The question therefore arises as to whether cues from the shopper’s other senses can be used to facilitate, guide, or bias visual search toward a particular product or product type. Prior research suggests that characteristic sounds can facilitate visual object localization (Iordanescu et al., 2008, 2010). Extending these findings to an applied setting, we investigated whether product-related sounds would facilitate visual search for products from different categories (e.g., champagne, potato crisps, deodorant) when arranged on a virtual shelf. On each trial, participants were visually presented with the name of a target product and then located the target within a virtual shelf display containing pictures of four different products (randomly selected from a set of nine). The visual display was randomly accompanied by a target-congruent, a target-incongruent, an unrelated, or no sound. Congruent sounds were semantically related to the target (e.g., uncorking a champagne bottle), incongruent sounds were related to the product shown in the corner opposite to the target, and unrelated sounds did not correspond to any of the products shown in the display. Participants found the target product significantly faster when the sound was congruent rather than incongruent with the target. All other pairwise comparisons were non-significant. These results extend the facilitatory crossmodal effect of characteristic sounds on visual search performance described earlier to the more realistic context of a virtual shelf display, showing that characteristic sounds can crossmodally enhance the visual processing of actual products.

Induced Roelofs effect in reaching

Event Abstract Back to Event Induced Roelofs effect in reaching Bahareh Taghizadeh1 and Alexander Gail1, 2* 1 German Primate Center, Germany 2 Bernstein Center for Computational Neuroscience, Germany Goal-directed reaching requires accurate localization of the target object. But the localization of visual objects is prone to illusions induced by the visual context. In the induced Roelofs effect (IRE), the position of a task irrelevant visual object induces a shift in the localization of the visual target object. This is true when subjects have to indicate the position of the target object relative to an array of reference positions, e.g. by response keys or by pointing to it with delay. In contrast, when subjects in the same task indicate the position of the target object by pointing to it without instructed delay or by directly reaching towards it with or without delay, no IRE is induced [1]. This discrepancy was taken as evidence for separate visuospatial representations for direct sensorimotor processing compared to spatial cognitive processing [2]. Here we test if an IRE can also be induced for reaching movement towards the target stimulus, which would argue against such strictly separate processing. We asked human subjects to perform reaches towards visual target stimuli in the frontoparallel plane. Each trial started with a brief appearance of a reference array (RA) of five horizontally arranged boxes which indicated the potential target positions. Then the target and a surrounding task-irrelevant visual frame were flashed, followed by a decision array (DA) immediately after the cue/frame offset. The DA was identical to the RA, but could be placed at different positions on the screen. While keeping ocular fixation, subjects had to reach towards the DA box which corresponded to the RA box in which they had seen the flashed target. The results showed a reliable IRE for reaching. In our task the reach goal needed to be defined relative to a task-relevant object, i.e. in an object-based reference frame, while in previous experiments the reach goal was directly defined by the target objects. The results suggest that during object-based encoding of motor goal locations, the information of additional task-irrelevant objects can induce systematic mis-localizations of the reach goal. This finding argues against strictly separate visuospatial representations for direct sensorimotor processing compared to spatial cognitive processing.

Integrating Recognition and Action Through Task-Relevant Knowledge for Daily Assistive Humanoids

This paper describes the design of a humanoid robot system that can perform daily assistive tasks. The significant issue in realizing a daily assistive humanoid robot system is to integrate an action subsystem and a recognition subsystem. We have designed a task-relevant knowledge that referred from both the action and recognition subsystems. The knowledge-based subsystem contains not only the information on an environment and objects' shapes, but also represents manipulation and navigation-related knowledge, and that for object recognition and robot localization. Since motion generation and an object recognition procedure share the same knowledge base system, the robot is capable of planning motion while recognizing objects and vice versa. This capability increases the effectiveness and robustness of the system. Three vision-guided behavior controls are presented during the execution of an action: (i) a visual self-localization to recognize the position of the robot, (ii) a visual object localization to update the object location in the model world to generate behaviors and (iii) a visual behavior verification to confirm the success of the motion. Finally, we demonstrated a kitchen service task by multiple humanoid robots to prove the broad capability and applicability of the proposed representation. The knowledge descriptions required for this demonstration consist of 13 behaviors, six objects with manipulation and visual feature knowledge, three search areas for the recognition process and two task-relevant visual behavior verification knowledge bases. We found that the description required for a kitchen service task is rather simple compared to the complexity of the demo scenario.

Neck muscle vibration in full cues affects pointing

Vibration of the dorsolateral neck stimulates proprioceptors that are normally active during head movement; this induces a visual illusion of contralateral motion and displacement of a stationary target seen against a homogenous background. The spatial constancy explanation of the illusion argues that it occurs because information about head movement is necessary for accurate egocentric localization of visual objects. Accurate egocentric localization, in turn, is necessary for the success of object-directed motor action, but previous studies failed to find evidence that vibration affects pointing toward visual targets in a normally illuminated, structured field. Our goal was to provide this evidence. Vibration lasting 12 s was applied to either side of the neck while observers (N = 11) pointed at the visual target with an unseen hand. Vibration of the right side of dorsal neck in the illuminated visual field induced a 26-mm lateral bias in pointing responses in comparison to the vibration of the left side. We conclude that the mechanism that takes into account neck proprioceptive signals also operates in full cues. The pointing bias in full cues generally co-occurred with reported stationariness of the visual target, suggesting a conflict between cues used in perception of body-centric position used to guide action, which include neck proprioception, and those used in perception of motion, for which object-relative retinal information is sufficient.

Human brain activation elicited by the localization of sounds delivering at attended or unattended positions: an fMRI/MEG study

The visual meridian effect, as a consequence of programming the necessary ocular movements for visual object localization (Rizzolatti et al. 1987), was interpreted as an evidence of a supramodal network for visual attention to space, which may be critical for reorienting the focus of attention toward stimuli appearing at unattended locations. This hypothesis was confirmed in a behavioural study also for auditory and for bimodal audio–visual stimulation (Ferlazzo et al. 2001). The present study is aimed at observing: the cerebral circuit underling the auditory meridian effect; whether this circuit is related with the network involved in reorienting the focus of attention toward stimuli appearing at an unattended location; whether a functional temporal dynamics can be defined in these areas.

Robust sound localization using multi-source audiovisual information fusion

This paper illustrates the synergic advantages of a multi-modal sound localization system utilizing two cameras and a 3-element microphone array. The two cameras were used as part of a stereo feature-detection based visual object localization system, while the microphones were combined to produce a sound localization system incorporating a temporal power fusion (TPF) algorithm. The cameras and microphones were integrated using spatial likelihood functions (SLFs), which greatly simplifies the integration process. Test results show a significant improvement in the integrated vision and sound localization (IVSL) system's ability over that of the stand-alone microphone-array based sound localization system to accurately localize sound sources in low signal to noise situations. The IVSL system maintained an average error of 15 cm at signal-to-noise ratios as low as 0.5 dB.

Visual Object Localization through Vestibular and Neck Inputs. 1: Localization with Respect to Space and Relative to the Head and Trunk Mid-Sagittal Planes

Object localization in space signals in the absence of an external reference (visual, auditory, haptic) involves a signal of the head in space (vestibular). The present study asks whether signals of body position relative to the support surface (proprioceptive) are involved as well, by investigating the role of vestibular-neck interaction (dissociating head and trunk position). Normal human subjects saw a light spot (object) and continuously nulled displacement steps of the spot. They did so before and after vestibular and/or neck rotational stimulation in the horizontal plane, reproducing a predesignated object localization in space (i), relative to the head mid-sagittal (ii), and relative to the trunk mid-sagittal (iii). The predominant frequency contained in the stimuli was varied (0.05, 0.1, and 0.4 Hz). (I) Object localization in space upon whole-body rotation (vestibular stimulus) at high frequency was veridical, whereas that at low frequency fell short. Almost identical results were obtained for trunk rotation about the stationary head (neck stimulus). In contrast, when combining the stimuli in the form of head rotation on the stationary trunk, the results were veridical, independent of stimulus frequency. Additional findings obtained with a large variety of vestibular-neck stimulus combinations suggest a linear summation of vestibular and neck signals. (II) Object localization with respect to the head was approximately veridical, being independent of vestibular and neck stimulation. However, this only applied if subjects were not biased by a head-in-space motion illusion of neck origin. (III) Object localization with respect to the trunk was veridical in all conditions tested. The findings support a recently developed concept, according to which humans evaluate the kinematic state of a visual object in space by (a) relating it to that of the body support by means of an essentially ideal proprioceptive coordinate transformation, and (b) relating, in turn, the kinematic state of the support to a vestibularly derived notion of space, using a proprioceptive coordinate transformation that "knows" the vestibular transfer characteristics. One important aspect is that object localization in space always is veridical during head and trunk rotation relative to a stationary support (for example, the ground) despite non-ideal vestibular transfer characteristics. Additional findings in patients with chronic loss of vestibular function confirm this concept.

Visual Object Localization through Vestibular and Neck Inputs. 2: Updating Off-Mid-Sagittal-Plane Target Positions

The vestibular signal plays a significant role in sensing changes in head orientation during rotations and in determining the magnitude of the rotations, but has only minor contributions in updating the internal representation of object positions with respect to the body after body rotations. The small contribution of the vestibular signal in egocentric object localization was evidenced in experiments in which the subjects reported the remembered position of eccentric earth-fixed targets after passive body rotations. The experiment reported here tested whether motor systems, such as the oculomotor system, make use of vestibular signals to generate accurate goal-directed motor responses toward a target whose position needs to be updated with respect to the body during and after whole-body rotations. The results showed that although subjects can produce saccadic eye movements of about the same magnitude as passive whole-body rotations (as previously reported by a number of researchers), they failed to generate accurate saccades toward the position of an extinguished peripheral visual target after the rotation. Overall, these results combined with those found in the literature suggest different central processes for determining changes in body orientation in complete darkness and for updating a target position with respect to the body during and after body rotations.

Perceived location of two-dimensional patterns

The study attempted to test the possibility that the center of gravity of two-dimensional patterns is the cue used by a human observer for their localization. Four experiments were carried out. The first, using a matching procedure, required the localization of the center of irregular dot patterns, contour and filled polygons which varied in size and orientation. In the other three experiments the subjects had to point to briefly exposed dot patterns in which overall shape (convex and concave in Expts 2 and 3) and dot density (Expt 4) were manipulated. The performance of these direct localization tasks was found to be as accurate as die performance in previous studies of indirect localization or regular patterns. The results consistently supported the claim that information about position of the center of gravity is used for the localization of visual objects.