Advanced Vision Systems Research Articles

Fruit picking robots: has their time come?

PurposeThe purpose of this paper is to describe recent fruit picking robot developments with an emphasis on corporate activity rather than academic research. It also aims to provide a view on the commercial prospects for these developments.Design/methodology/approachFollowing a short introduction, this first discusses strawberry and other soft fruit picking robot developments conducted principally by commercial organisations. It then provides similar details of robots for harvesting apples and other hard fruits. This is followed by a discussion and concluding comments.FindingsThe shortage of seasonal fruit pickers has stimulated the need for automation. Accordingly, a growing community of companies, many founded in the past five years, are developing fruit picking robots. These are aimed at both soft and hard fruits, such as strawberries and apples, respectively, and exploit advanced vision systems, image processing techniques and AI. Some products are already on the market, whereas many more are due for commercial release during the next two years into what is expected to be a highly competitive market.Originality/valueThis provides details of the emerging fruit picking robot business by describing the products and manufacturing companies.

An Optoneuronic Device with Realistic Retinal Expressions for Bioinspired Machine Vision

Machine vision systems rely on communication between cameras and processor modules to capture and analyze visual information. This arrangement renders them as bulky and inefficient in terms of speed and power dissipation for futuristic big data applications that involve artificial intelligence algorithms. An apparatus able to imitate the operation of biologic eyes and function as a standalone platform would therefore present the next evolutional step in machine visual perception. Neuromorphic computing is an alternative approach to the Von Neumann architecture that carries the potential for implementing such intelligent cameras. In this regard, artificial synaptic devices have been widely used in recent years to construct hardware‐based neural networks mainly due to their adjustable electric parameters. Herein, a bioinspired, hybrid electrophotonic responsive neuronic device that mimics the combined functionality of retinal cones and bipolar cells is demonstrated. Under illumination, it features a hyperpolarization‐like current response in an OFF state and a complementary depolarized reaction when toggled to an ON state. Furthermore, electrical pulsing done in conjunction with light stimulation can emulate the horizontal cell neurotransmitter release in center‐surround biologic configurations. These devices may thus serve as building blocks for advanced visual systems, integrating self‐healing sensory and neuromorphic computing into an artificial cognitive retina.

Multifunctional Liquid Marble Compound Lenses.

Mosquito compound eyes are elaborate multifunctional hierarchical structures. The presence of ordered curved features spanning length scales of nanometers to millimeters provides the mosquito eye with a wide field of view, an infinite depth of field, and antifogging properties. Developing bio-inspired compound lenses is challenging because of the need to mimic all characteristic curvatures along with their functionalities. Herein, we show how the curvature inherent to nanoparticles, emulsion droplets, and liquid marbles can be employed to mimic the hierarchical structure and functionality of mosquito compound eyes. At the nanometer to micrometer length scale we employ nanoparticle-stabilized emulsion droplets of photocurable oil to form microlenses with nanoscale surface features. After polymerization, the microlenses form a monolayer on an oil droplet to create an optically clear, millimeter scale, liquid marble that functions as a compound lens. We characterize the optical and surface properties of the compound lenses and find that they reproduce the functionality of the mosquito eye. Additionally, we exploit the mobility and reconfigurability of liquid marbles to create arrays (centimeter scale) of compound lenses and other types of functional lenses such as the Janus lens that magnifies the image acquired by the compound lens. Simple and scalable methods to create compound lenses could aid in the development of miniaturized advanced vision systems.

Scaffold-free bioprinting of mesenchymal stem cells with the regenova printer: Optimization of printing parameters

The Kenzan bioprinting method provides a high-resolution biofabrication process by facilitating the fusion of submillimeter cell aggregates (spheroids) into larger tissue constructs on a needle array that is removed upon spheroid fusion. Although the method is relatively straightforward in principle, Kenzan method bioprinting relies on a complex 3D bioprinter (Regenova Bio 3D Printer, Cyfuse, K.K., Japan) implementing an advanced vision system to verify the microscopic spheroids’ geometry and high-precision mechatronics to aseptically manipulate the spheroids into position. Due to the complexity of the operation, the need for aseptic conditions, and the size of the spheroids, proficiency with the Regenova Bio 3D Printer and the Kenzan method requires development of best practices and troubleshooting techniques to ensure a robust print and minimize the use of resources. In addition, managing the construct post-bioprinting both in culture and for surgical implantation requires careful consideration and workflow design. Here, we describe methods for generating a competent tissue construct and optimizing the bioprinting process. Optimization resulted in a 4-fold reduction in print times, a 20-fold reduction in the use of bioprinting nozzles, and more robust constructs. The results and procedures described herein will have potential applications for tissue engineering, research, and clinical uses in the future.

Miniaturising artificial compound eyes based on advanced micronanofabrication techniques

Compound eyes (CEs) are advanced optical visual systems with distinct features of large view-fields, infinite depth of field, and dynamic imaging capability, revealing their significant potential in applications including robot vision, unmanned aerial vehicle detection, and medical diagnosis. Compared with macroscopic CEs, which primarily comprise multicamera arrays, compact integrated CEs have garnered significant attention because of their portability and possibility of flexible integration with microrobots and in-vivo medical facilities. To date, considerable effort has been devoted to this field, in which manufacturing technologies are vital to the development of artificial CEs (ACEs) capable of large field-of-view imaging, depth information collection, and three-dimensional imaging. Challenges and opportunities exist for the practical application of advanced ACEs. This paper reviews state-of-the-art technologies for manufacturing ACEs, and then briefly summarises their potential applications in different fields. Finally, the current challenges and perspectives of ACEs are discussed.

Underwater navigation using visual markers in the context of intervention missions

Intervention missions, that is, underwater manipulation tasks, for example, in the context of oil-&-gas production, require a high amount of precise, robust navigation. In this article, we describe the use of an advanced vision system suited for deep-sea operations, which in combination with artificial markers on target structures like oil-&-gas production-Christmas-trees significantly boosts navigation performance. The system is validated in two intensive field tests running off the shore of Marseille, France. In the experiments, a commercial remotely operated vehicle equipped with the system and a mock-up structure with an oil-&-gas production panel is used to evaluate the navigation performance.

Biosensores y sistemas ópticos y de visión avanzados: su aplicación en la evaluación de la calidad de productos IV gama: su aplicación en la evaluación de la calidad de productos IV gama

Los productos mínimamente procesados son alimentos muy atractivos para los consumidores atendiendo a sus propiedades nutricionales y a su facilidad de preparación. Sin embargo, la seguridad microbiológica de estos alimentos requiere de especial atención, dado que a lo largo de su proceso no se someten a tratamientos que aseguren la letalidad de los microorganismos. Aparece así la necesidad de dotar a la industria alimentaria en general, y a la de IV gama en particular, de pruebas rápidas para la detección de contaminantes. En este trabajo se presentan los fundamentos de los biosensores y la imagen hiperespectral (IH) y sus aplicaciones en la aportación de información que se pueda correlacionar con la calidad y el estado de frescura de los productos IV gama y de otros alimentos.

Convergent evolution of tertiary structure in rhodopsin visual proteins from vertebrates and box jellyfish

Box jellyfish and vertebrates are separated by >500 million years of evolution yet have structurally analogous lens eyes that employ rhodopsin photopigments for vision. All opsins possess a negatively charged residue-the counterion-to maintain visible-light sensitivity and facilitate photoisomerization of their retinaldehyde chromophore. In vertebrate rhodopsins, the molecular evolution of the counterion position-from a highly conserved distal location in the second extracellular loop (E181) to a proximal location in the third transmembrane helix (E113)-is established as a key driver of higher fidelity photoreception. Here, we use computational biology and heterologous action spectroscopy to determine whether the appearance of the advanced visual apparatus in box jellyfish was also accompanied by changes in the opsin tertiary structure. We found that the counterion in an opsin from the lens eye of the box jellyfish Carybdea rastonii (JellyOp) has also moved to a unique proximal location within the transmembrane bundle-E94 in TM2. Furthermore, we reveal that this Schiff base/counterion system includes an additional positive charge-R186-that has coevolved with E94 to functionally separate E94 and E181 in the chromophore-binding pocket of JellyOp. By engineering this pocket-neutralizing R186 and E94, or swapping E94 with the vertebrate counterion E113-we can recreate versions of the invertebrate and vertebrate counterion systems, respectively, supporting a relatively similar overall architecture in this region of animal opsins. In summary, our data establish the third only counterion site in animal opsins and reveal convergent evolution of tertiary structure in opsins from distantly related species with advanced visual systems.

Non-invasive in vivo quantification of the developing optical properties and graded index of the embryonic eye lens using SPIM.

Graded refractive index lenses are inherent to advanced visual systems in animals. By understanding their formation and local optical properties, significant potential for improved ocular healthcare may be realized. We report a novel technique measuring the developing optical power of the eye lens, in a living animal, by exploiting the orthogonal imaging modality of a selective plane illumination microscope (SPIM). We have quantified the maturation of the lenticular refractive index at three different visible wavelengths using a combined imaging and ray tracing approach. We demonstrate that the method can be used with transgenic and vital dye labeling as well as with both fixed and living animals. Using a key eye lens morphogen and its inhibitor, we have measured their effects both on lens size and on refractive index. Our technique provides insights into the mechanisms involved in the development of this natural graded index micro-lens and its associated optical properties.

Preoperative localization of parathyroid glands. Use of MRI, scintigraphy, and image fusion.

Minimally invasive resection of hyperfunctional parathyroid glands is an alternative to open surgery. However, it requires a precise preoperative localization. This study evaluated the diagnostic use of magnetic resonance (MR) imaging, parathyroid scintigraphy, and consecutive image fusion. 17 patients (9 women, 8 men; age: 29-72 years; mean: 51.2 years) with primary hyperparathyroidism were included. Examination by MRI used unenhanced T1- and T2-weighted sequences as well as contrast-enhanced T1-weighted sequences. (99m)Tc-MIBI scintigraphy consisted of planar and SPECT (single photon emission tomography) imaging techniques. In order to improve the anatomical localization of a scintigraphic focus, SPECT-data were fused with the corresponding MR-data using a modified version of the Express 5.0 software (Advanced Visual Systems, Waltham, MA). Results of image fusion were then compared to histopathology. In 14/17 patients, a single parathyroid adenoma was found. There were 3 cases with hyperplastic glands. MRI detected 10 (71%), scintigraphy 12 (86%) adenomas. Both modalities detected 1/3 patients with hyperplasia. Image fusion improved the anatomical assignment of the 13 scintigraphic foci in five patients and was helpful in the interpretation of inconclusive MR-findings in two patients. Both MRI and (99m)Tc-MIBI scintigraphy sensitively detect parathyroid adenomas but are less reliable in case of hyperplastic glands. In case of a scintigraphic focus, image fusion considerably improves its topographic assignment. Furthermore, it facilitates the evaluation of inconclusive MRI findings.

An Advanced Active Vision System with Multimodal Visual Odometry Perception for Humanoid Robots

Using active vision to perceive surroundings instead of just passively receiving information, humans develop the ability to explore unknown environments. Humanoid robot active vision research has already half a century history. It covers comprehensive research areas and plenty of studies have been done. Nowadays, the new trend is to use a stereo setup or a Kinect with neck movements to realize active vision. However, human perception is a combination of eye and neck movements. This paper presents an advanced active vision system that works in a similar way as human vision. The main contributions are: a design of a set of controllers that mimic eye and neck movements, including saccade eye movements, pursuit eye movements, vestibulo-ocular reflex eye movements and vergence eye movements; an adaptive selection mechanism based on properties of objects to automatically choose an optimal tracking algorithm; a novel Multimodal Visual Odometry Perception method that combines stereopsis and convergence to enable robots to perform both precise action in action space and scene exploration in personal space. Experimental results prove the effectiveness and robustness of our system. Besides, the system works in real-time constraints with low-cost cameras and motors, providing an affordable solution for industrial applications.

ViPS: A novel visual processing system architecture for medical imaging

Abstract Imaging has become an essential tool in modern medicine science. Numerous powerful platforms to register, store, analyze and process medical imaging applications appear in recent years. In this article, we have designed an advanced visual processing system (ViPS) that stores and processes complex and multi-dimensional medical imaging application. The ViPS provides a user-friendly programming environment and high-performance architecture for data acquisition, registration, storage, analysis and performs segmentation, filtering, and recognition of complex real-time complex and multidimensional medical images or videos. The proposed architecture is highly reliable concerning cost, performance, and power. ViPS is designed and evaluated on a Xilinx Virtex-7 FPGA VC707 Evaluation Kit. The performance of ViPS is compared with the Heterogeneous Multi-Processing Odroid XU3 board and GPU Jetson TK1 Embedded Development Kit with 192 CUDA cores based graphic systems. In comparison to the Heterogeneous Multi-core and GPU based Graphics systems, the results show that ViPS improves 2.4 and 1.4 times system performance respectively, for iridology application. While executing real-time complex images reconstruction at 2× and 1.25× of higher frame rate, the ViPS achieves 15.2× and 5.26× of performance improvement while running various image processing algorithms. The ViPS gets 3.01× and 1.13× of speedup for video processing algorithms and draws 1.55 and 2.27 times less dynamic power.

Toward Theoretical Techniques for Measuring the Use of Human Effort in Visual Analytic Systems.

Visual analytic systems have long relied on user studies and standard datasets to demonstrate advances to the state of the art, as well as to illustrate the efficiency of solutions to domain-specific challenges. This approach has enabled some important comparisons between systems, but unfortunately the narrow scope required to facilitate these comparisons has prevented many of these lessons from being generalized to new areas. At the same time, advanced visual analytic systems have made increasing use of human-machine collaboration to solve problems not tractable by machine computation alone. To continue to make progress in modeling user tasks in these hybrid visual analytic systems, we must strive to gain insight into what makes certain tasks more complex than others. This will require the development of mechanisms for describing the balance to be struck between machine and human strengths with respect to analytical tasks and workload. In this paper, we argue for the necessity of theoretical tools for reasoning about such balance in visual analytic systems and demonstrate the utility of the Human Oracle Model for this purpose in the context of sensemaking in visual analytics. Additionally, we make use of the Human Oracle Model to guide the development of a new system through a case study in the domain of cybersecurity.

Advanced Vision System by Subtraction of Background Image for Patient’s Movement during Treatment

According to the International Cancer Institute (IARC), the incidence of cancer in the world is rapidly increasing, and in the method of treating cancer, radiation therapy is 45% or more. As in the normal tissue in radiation therapy, to minimize radiation, it is important to maximize the exact dose to the tumor sites for local tumor control rates in intensive research (Local Tumor Rate). Therefore, at the beginning, therapist accuracy of detecting fatigue in the movement of the patient and therapist has been a problem that is weighted down directly. Also, by using a web camera, a difference value between the image to be updated to the reference image is calculated, and if the result exceeds the reference value, using the system for determining the motion has occurred. However, it is not possible in this system to quantitatively analyze the movement of the patient, as the background is changed when moving the treatment bed in the co-therapeutic device was not able to shift the patient. In this paper, using an alpha (α) filter index is an attempt to solve these limitation points, as it quantifies the movement of the patient by separating a background image of the patient and treatment environment, and it only senses movement of the patient during treatment, it was possible to reduce the problems due to patient movement.

Adaptation of the Mitochondrial Genome in Cephalopods: Enhancing Proton Translocation Channels and the Subunit Interactions.

Mitochondrial protein-coding genes (mt genes) encode subunits forming complexes of crucial cellular pathways, including those involved in the vital process of oxidative phosphorylation (OXPHOS). Despite the vital role of the mitochondrial genome (mt genome) in the survival of organisms, little is known with respect to its adaptive implications within marine invertebrates. The molluscan Class Cephalopoda is represented by a marine group of species known to occupy contrasting environments ranging from the intertidal to the deep sea, having distinct metabolic requirements, varied body shapes and highly advanced visual and nervous systems that make them highly competitive and successful worldwide predators. Thus, cephalopods are valuable models for testing natural selection acting on their mitochondrial subunits (mt subunits). Here, we used concatenated mt genes from 17 fully sequenced mt genomes of diverse cephalopod species to generate a robust mitochondrial phylogeny for the Class Cephalopoda. We followed an integrative approach considering several branches of interest–covering cephalopods with distinct morphologies, metabolic rates and habitats–to identify sites under positive selection and localize them in the respective protein alignment and/or tridimensional structure of the mt subunits. Our results revealed significant adaptive variation in several mt subunits involved in the energy production pathway of cephalopods: ND5 and ND6 from Complex I, CYTB from Complex III, COX2 and COX3 from Complex IV, and in ATP8 from Complex V. Furthermore, we identified relevant sites involved in protein-interactions, lining proton translocation channels, as well as disease/deficiencies related sites in the aforementioned complexes. A particular case, revealed by this study, is the involvement of some positively selected sites, found in Octopoda lineage in lining proton translocation channels (site 74 from ND5) and in interactions between subunits (site 507 from ND5) of Complex I.

Effective Calibration and Evaluation of Multi-Camera Robotic Head

The paper deals with appropriate calibration of multispectral vision systems and evaluation of the calibration and data-fusion quality in real-world indoor and outdoor conditions. Checkerboard calibration pattern developed by our team for multispectral calibration of intrinsic and extrinsic parameters is described in detail. The circular object for multispectral fusion evaluation is described as well. The objects were used by our team for calibration and evaluation of advanced visual system of Orpheus-X3 robot that is taken as a demonstrator, but their use is much wider, and authors suggest to use them as testbed for visual measurement systems of mobile robots. To make the calibration easy and straightforward, the authors developed MultiSensCalib program in Matlab, containing all the described techniques. The software is provided as publicly available, including source code and testing images.

An Advanced Vision System for the Automatic Inspection of Corrosions on Pole Tips in Hard Disk Drives

Pole tips of hard disk drives are made of FeCo film, which is prone to corrosion due to the corrosive nature of the environment and wide range of pH experienced during the production. A machine vision system is utilized for the automatic auditing of corroded pole tips. This comprises three steps: 1) extraction of the top shield region; 2) fusion of extracted features; and 3) decision making. This model is capable of detecting corrosion in the critical part of the pole tips. The experiments show that the proposed system detects corrosion with high accuracy, whereas the overall processing time meets industrial requirements.

De novo transcriptome of Ischnura elegans provides insights into sensory biology, colour and vision genes.

BackgroundThere is growing interest in odonates (damselflies and dragonflies) as model organisms in ecology and evolutionary biology but the development of genomic resources has been slow. So far only one draft genome (Ladona fulva) and one transcriptome assembly (Enallagma hageni) have been published. Odonates have some of the most advanced visual systems among insects and several species are colour polymorphic, and genomic and transcriptomic data would allow studying the genomic architecture of these interesting traits and make detailed comparative studies between related species possible. Here, we present a comprehensive de novo transcriptome assembly for the blue-tailed damselfly Ischnura elegans (Odonata: Coenagrionidae) built from short-read RNA-seq data. The transcriptome analysis in this paper provides a first step towards identifying genes and pathways underlying the visual and colour systems in this insect group.ResultsIllumina RNA sequencing performed on tissues from the head, thorax and abdomen generated 428,744,100 paired-ends reads amounting to 110 Gb of sequence data, which was assembled de novo with Trinity. A transcriptome was produced after filtering and quality checking yielding a final set of 60,232 high quality transcripts for analysis. CEGMA software identified 247 out of 248 ultra-conserved core proteins as ‘complete’ in the transcriptome assembly, yielding a completeness of 99.6%. BLASTX and InterProScan annotated 55% of the assembled transcripts and showed that the three tissue types differed both qualitatively and quantitatively in I. elegans. Differential expression identified 8,625 transcripts to be differentially expressed in head, thorax and abdomen. Targeted analyses of vision and colour functional pathways identified the presence of four different opsin types and three pigmentation pathways. We also identified transcripts involved in temperature sensitivity, thermoregulation and olfaction. All these traits and their associated transcripts are of considerable ecological and evolutionary interest for this and other insect orders.ConclusionsOur work presents a comprehensive transcriptome resource for the ancient insect order Odonata and provides insight into their biology and physiology. The transcriptomic resource can provide a foundation for future investigations into this diverse group, including the evolution of colour, vision, olfaction and thermal adaptation.Electronic supplementary materialThe online version of this article (doi:10.1186/1471-2164-15-808) contains supplementary material, which is available to authorized users.

Performance Evaluation of a Low-Cost Stereo Vision System for Underwater Object Detection

This paper describes the development and performance assessment of a low-cost stereo vision system for underwater object detection. The system has been conceived as a prototype to investigate the performance, power consumption, and thermal dissipation tradeoffs involved in designing an embedded stereo vision unit for underwater operation. The embedded system has been experimentally assessed in underwater object detection tasks. The system has proven thermally stable and capable of guaranteeing a level of autonomy of at least two hours of video acquisition. Several algorithms for mono and stereo image processing have been evaluated to assess their effectiveness in the underwater environment along with their suitability in presence of constrained computational and energy resources. Evaluation of the stereo vision system in detecting simple objects has shown strong limitations of commodity off-the-shelf sensors when used in underwater perception. Nevertheless, the prototype described in this work provides insights for development of more advanced vision systems suitable for underwater vehicles.

Visual Reconstruction and Analysis of the Surface of Earthworm in AVS

This paper describes a method for visual reconstruction and analysis of the surface of earthworm in AVS/Express, Advanced Visual System's new visualization development tool. The digital measurements of surfaces of the earthworm were carried out using a three-dimensional laser scanner. Detailed information on the point clouds, the scanning digital data of the surface of the earthworm were obtained by importing and analyzing those point clouds in AVS/Express. Methods and results presented in this paper proved to be potentially useful for analyzing the feature of biological prototype, optimizing the mathematical model and affording deformable physical model to bionic engineering, this would have significant implications to the research of biological coupling theory and technological creation in bionic engineering.