Smart Vision System Research Articles

AdvRain: Adversarial Raindrops to Attack Camera-Based Smart Vision Systems

Vision-based perception modules are increasingly deployed in many applications, especially autonomous vehicles and intelligent robots. These modules are being used to acquire information about the surroundings and identify obstacles. Hence, accurate detection and classification are essential to reach appropriate decisions and take appropriate and safe actions at all times. Current studies have demonstrated that “printed adversarial attacks”, known as physical adversarial attacks, can successfully mislead perception models such as object detectors and image classifiers. However, most of these physical attacks are based on noticeable and eye-catching patterns for generated perturbations making them identifiable/detectable by the human eye, in-field tests, or in test drives. In this paper, we propose a camera-based inconspicuous adversarial attack (AdvRain) capable of fooling camera-based perception systems over all objects of the same class. Unlike mask-based FakeWeather attacks that require access to the underlying computing hardware or image memory, our attack is based on emulating the effects of a natural weather condition (i.e., Raindrops) that can be printed on a translucent sticker, which is externally placed over the lens of a camera whenever an adversary plans to trigger an attack. Note, such perturbations are still inconspicuous in real-world deployments and their presence goes unnoticed due to their association with a natural phenomenon. To accomplish this, we develop an iterative process based on performing a random search aiming to identify critical positions to make sure that the performed transformation is adversarial for a target classifier. Our transformation is based on blurring predefined parts of the captured image corresponding to the areas covered by the raindrop. We achieve a drop in average model accuracy of more than 45% and 40% on VGG19 for ImageNet dataset and Resnet34 for Caltech-101 dataset, respectively, using only 20 raindrops.

TOWARDS REMOTE CONTROL OF MANUFACTURING MACHINES THROUGH ROBOT VISION SENSORS

AbstractThe remote management of equipment is part of the functionalities granted by the design principles of Industry 4.0. However, some critical operations are managed by operators, machine setup and initialization serve as a significant illustration. Since the initialization is a repetitive task, industrial robots with a smart vision system can undertake these duties, enhancing the autonomy and flexibility of the manufacturing process. The smart vision system is considered essential for the implementation of several characteristics of Industry 4.0. This paper introduces a novel solution for controlling manufacturing machines using an embedded camera on the robot. This implementation requires the development of an interactive interface, designed in accordance with the supervision system known as Manufacturing Execution System. The framework is implemented inside a manufacturing cell, demonstrating a quick response time and an improvement between the cameras.

Development and Evaluation of a Small-Scale Apple Sorting Machine Equipped with a Smart Vision System

One of the most important matters in international trades for many local apple industries and auctions is accurate fruit quality classification. Defect recognition is a key in online computer-assisted apple sorting machines. Because of the cavity structure of the stem and calyx regions, the system tends to mistakenly treat them as true defects. Furthermore, there is no small-scale sorting machine with a smart vision system for apple quality classification where it is needed. Thus, the current study focuses on a highly accurate and feasible methodology for stem and calyx recognition based on Niblack thresholding and a machine learning technique using k-nearest neighbor (k-NN) classifiers associated with a locally designed small-scale apple sorting machine. To find an appropriate mode, the effects of different numbers of k and metric distances on stem and calyx region detection were evaluated. Results showed the effectiveness of the value of k and Euclidean distances in recognition accuracy. It is found that the 5-nearest neighbor classifier and the Euclidean distance using 80 training samples produced the best accuracy rates, at 100% for stem and 97.5% for calyx. The significance of the result is very promising in fabricating an advanced small-scale and low-cost sorting machine with a high accuracy for the horticultural industry.

A Modular Smart Vision System for Industrial Inspection and Control of Conformity

Abstract In manufacturing systems, there are several essential tasks to perform before and after the production process. In traditional systems, these tasks were done manually, which can lead to more resource consumption and the risk of human error; however, advanced manufacturing systems and Industry 4.0 tend toward a more autonomous manner. To ensure the compliance of the machining process and the safety of the personnel as well as the machines at the shop floor, the inspection of the overall factory prior to any machining process and the control of conformity of the manufactured parts are necessary in order to know the status of the manufacturing line. This paper proposes a novel modular smart vision system for machine inspection and conformity control of machined parts. Our system uses smart vision technologies embedded in industrial robots and enhanced with image processing and analysis capabilities. The solution also integrates a user interface for human–machine interactions that has been developed with a modular approach, and is designed, launched, and controlled by the manufacturing execution system, allowing agile and customized configuration. By this new approach, the robot inspects all the machines in the factory to check the status before launching the production plan. After the machining process, the system interprets the in situ dimensional analysis for the machined parts and makes decisions about whether the parts are acceptable or require additional machining.

Wireless SmartVision system for synchronized displacement monitoring of railroad bridges

AbstractThe deflection of railroad bridges under in‐service loads is an important indicator of the structure's health. Over the past decade, an increasing number of studies have demonstrated the efficacy of using vision‐based approaches for displacement tracking of civil infrastructure. These studies have relied primarily on external processing of manually recorded videos of a structure's motion to estimate displacements. To date, vision‐based techniques applied to long‐term structural health monitoring have yet to be proven effective as an alternative to the traditional displacement measurement methods, such as linear variable differential transformers. This paper proposes a wireless SmartVision system (WSVS) that uses edge computing to directly output bridge displacements that can be sent to the end user. The system estimates displacements using both target‐free and target‐based approaches. A synchronized sensing framework is developed for multipoint displacement estimation using several wireless vision‐based nodes for full‐scale displacement‐based modal analysis of structures. Pose estimation using an AprilTag, a fiducial marker, is employed with a modified algorithm for improved displacement tracking of targets installed on a bridge, yielding subpixel accuracy. The robustness of the results in field conditions is enhanced by linking a tracking quality factor to each timestamp to handle vision‐related uncertainties. To meet the need for precise error metrics evaluation, an inexpensive cyber‐physical setup using a synthetic testing environment is also developed in this study. Following laboratory validation, field tests on a cable‐stayed pedestrian bridge were performed to demonstrate the efficacy of the proposed WSVS.

Influence of technological advances and change in marketing strategies using analytics in retail industry

A nexus of technological advances and an increasingly competitive environment of the retail industry has taken the phrase, “Customer is the King” to a new tangent altogether. It has been observed how combination of technologies along with analytical concepts of video analytics, social media analytics, wireless analytics and smart vision systems on marketing concepts like market basket model, value-based customer segmentation, campaign planning, etc. can impact the customer satisfaction and reduce the customer churn rate. An effective amalgamative implementation of these concepts will help enhance customer satisfaction and help the retailers gain an edge in the competitive market environment. The aim of this paper is to understand the technological advancements along with the impact of data analytics in the retail sector and to capture and retain maximum customers by conceptualizing effective merchandising and marketing strategies.

On building a CNN-based multi-view smart camera for real-time object detection

Multi-view image sensing is currently gaining momentum, fostered by new applications such as autonomous vehicles and self-propelled robots. In this paper, we prototype and evaluate a multi-view smart vision system for object recognition. The system exploits an optimized Multi-View Convolutional Neural Network (MVCNN) in which the processing is distributed among several sensors (heads) and a camera body. The main challenge for designing such a system comes from the computationally expensive workload of real-time MVCNNs which is difficult to support with embedded processing and high frame rates. This paper focuses on the decisions to be taken for distributing an MVCNN on the camera heads, each camera head embedding a Field-Programmable Gate Array (FPGA) for processing images on the stream. In particular, we show that the first layer of the AlexNet network can be processed at the nearest of the sensors, by performing a Direct Hardware Mapping (DHM) using a dataflow model of computation. The feature maps produced by the first layers are merged and processed by a camera central processing node that executes the remaining layers. The proposed system exploits state-of-the-art deep learning optimization methods, such as parameter removing and data quantization. We demonstrate that accuracy drops caused by these optimizations can be compensated by the multi-view nature of the captured information. Experimental results conducted with the AlexNet CNN show that the proposed partitioning and resulting optimizations can fit the first layer of the multi-view network in low-end FPGAs. Among all the tested configurations, we propose 2 setups with an equivalent accuracy compared to the original network on the ModelNet40 dataset. The first one is composed of 4 cameras based on a Cyclone III E120 FPGA to embed the least expensive version in terms of logic resources while the second version requires 2 cameras based on a Cyclone 10 GX220 FPGA. This distributed computing with workload reduction is demonstrated to be a practical solution when building a real-time multi-view smart camera processing several frames per second.

Distributed mixed-signal architecture for programmable smart image sensors

Smart vision systems on a chip are promising for embedded applications. Currently, flexibility in the choice of integrated pre-processing tools is obtained at the expense of total silicon area and fill factor, which are otherwise optimized provided that the sensor performs a specific task. We propose a new architecture based on macropixel-level processing to improve the trade-off by using the same processing elements (PEs) for a whole group of pixels. In this paper, we show through transistor-level simulations the feasibility of using macropixel PEs. Their operative part is analog to avoid the bottleneck of analog to digital converters and has digital control which is distributed in and out of the matrix of pixels. PEs are designed to be suitable for coefficient-reconfigurable spatial and temporal filtering. Sharing electronics among several pixels and matching existing algorithms to the target architecture allow for such programmability without degrading too much pixel area nor fill factor.

Reflective workpiece detection and localization for flexible robotic cells

A smart vision system for industrial robotic cells is presented. It can recognize and localize a reflective workpiece, and allows for automatic adjustments of the robot program. The purpose of the study is to improve industrial robots awareness of the environment and to increase adaptability of the manufacturing processes where full control over environment is not achievable. This approach is particularly relevant to small batch robotic production, often suffering from only partial control of the process parameters, such as the order of jobs, workpiece position, or illumination conditions.A distinguishing aspect of the study is detection of workpieces made of diverse materials, including shiny metals. Reflective surfaces are common in the industrial manufacturing, but are rarely considered in the research on object recognition because they hinder many of the object recognition algorithms. The proposed solution has been qualified and tested on a selected benchmark in a realistic workshop environment with artificial light conditions. The training of the object recognition software is an automatic process and can be executed by non-expert industrial users to allow for recognition of different types of objects. HighlightsA machine vision system for detection of workpieces in a robotic cell is proposed.The system can recognize objects with specular surfaces (made of reflective metal).It estimates the location of the object on the workbench from a single camera image.It adjusts the robot program according to the presence of the workpiece and its type.

Dataflow object detection system for FPGA‐based smart camera

Embedded computer vision based smart systems raise challenging issues in many research fields, including real-time vision processing, communication protocols or distributed algorithms. The amount of data generated by cameras using high resolution image sensors requires powerful computing systems to be processed at digital video frame rates. Consequently, the design of efficient and flexible smart cameras, with on-board processing capabilities, has become a key issue for the expansion of smart vision systems relying on decentralised processing at the image sensor node level. In this context, field programmable gate arrays (FPGA)-based platforms, supporting massive data parallelism, offer large opportunities to match real-time processing constraints compared with platforms based on general purpose processors. In this study, the authors describe the implementation, on such a platform, of a configurable object detection application, reformulated according to the dataflow model of computation. The application relies on the computation of the histogram of oriented gradients and a linear SVM-based classification. It is described using the CAPH programming language, allowing efficient hardware descriptions to be generated automatically from high level dataflow specifications without prior knowledge of hardware description languages such as VHDL or Verilog. Results show that the performance of the generated code does not suffer from a significant overhead compared with handwritten HDL code.

A Memory-Efficient Architecture of Full HD Around View Monitor Systems

The around view monitor (AVM) is one of the representative features of smart vision systems adopted in various application areas, e.g., advanced driver assistance systems. The design of AVM systems with full high-definition (HD)-level resolution presents significant technical challenges. In particular, a high-memory performance is required to process full HD images obtained from multiple cameras. Specifically, a full HD AVM system requires a high-memory bandwidth (six times higher than D1 image-based systems) and is characterized by a significant amount of single writes, which degrades the effective performance of modern DRAM. To address these problems, two methods are proposed in this paper. The first method reduces the required memory bandwidth by storing in the memory only the input pixel data that will be used in the final processing step, and the second improves the single-write performance of a DRAM subsystem by DRAM-aware data mapping. The effectiveness of the proposed methods is proved by designing an AVM system incorporating field-programmable gate arrays and DDR2-200 SDRAMs. The proposed methods reduce the memory bandwidth requirement by 51%, allowing a full HD AVM system to run at over 24 frames per second.

Providing accessibility to blind people using GIS

In everyday life, people need to move, whether in business or leisure. Navigation requires spatial knowledge and ability to make decisions based on geographic information. Recently, powerful tools have been developed, enhancing the capabilities of geographical analysis and decision-making. This work presents a platform to handle and provide geographic information, including accessibility oriented features. This geographic information system (GIS) is part of a wider project, called SmartVision. The aim of this project is to create a system that allows blind users to navigate in the University of Tras-os-Montes and Alto Douro campus. The GIS platform, together with other modules of the SmartVision system prototype, provides information to blind users, assisting their navigation and giving alerts of nearby points-of-interest or obstacles. Together with the GIS platform, this paper also describes the handling of accessibility information by the SmartVision prototype, namely the Navigation Module, the Computer Vision Module and the Interface Module.

Multi-modal defect detection of residual oxide scale on a cold stainless steel strip

This work presents two initial approaches and a novel technique for the industrial inspection of residual oxide scale on a cold stainless steel strip. The research aims to develop real-time systems to detect 50-μm defects. Initially, a spectrophotometric analysis provides the wavelength regions where differences between stainless steel and residual oxide scale reflectance are highlighted. The multi-modal approach is based on laser techniques that comprise three different strategies to gradually achieve a robust stainless steel industrial inspection through the evaluation of their performance. First, an inspection system based on a single commercial laser has been designed with a dynamic threshold module. In the second approach, the inspection task is accomplished by volatilizing a reduced area of the stainless steel surface with short pulses of a high-power ultraviolet laser and then analyzing the generated plasma with an intensifier camera. The third technique consists of an innovative smart vision system for surface visual inspection based on laser diode diffuse illumination. This vision system can be configured to work with two laser illumination modes: the diffuse coaxial lighting and the diffuse bright-field lighting. These techniques aim to gradually improve surface defect detection of a cold stainless steel strip. Furthermore, some of the results of the defect detection level obtained with each approach are displayed and discussed.

Micro-observation technique for tele-micro-operation

In this paper, an observational sensor system for tele-micro-operation is proposed with a dynamic focusing lens and a smart vision sensor using the 'depth from focus' criteria. Recently, micro-operations, such as for micro-surgery, DNA manipulations, etc., have gained in importance. However, the small depth of focus of the microscope produces poor observability. For example, if the focus is on the object, the actuator cannot be seen with the microscope. On the other hand, if the focus is on the actuator, the object cannot be observed. In this sense, the 'all-in-focus image', which holds the in-focused texture all over the image, is useful to observe the micro-environments with a microscope. One drawback of the all-in-focus image is that there is no information about the depth of objects. It is also important to obtain the depth map and show the three-dimensional (3D) micro virtual environments in real-time to actuate the micro objects intuitively. First, this paper reviews the criteria of 'depth from focus' to achieve the all-in-focus image and the 3D micro environments' simultaneous reconstruction. After evaluating the validity of this criteria with off-line simulation, a real-time virtual reality (VR) micro camera system is proposed to achieve the micro VR environments with the 'depth from focus' criteria. This system is constructed with a dynamic focusing lens, which can change its focal distance at a high frequency, and a smart vision system, which is capable of capturing and processing the image data in high speed with SIMD architecture.

Implementation of Real Time Micro VR Camera

In this paper, a real time virtual reality camera system for the micro environments has been developed with a dynamic focusing lens and a smart vision sensor using the from criteria.In the operation with a microscope, such as the micro-surgery, DNA operation and etc., the small depth of a focus makes bad controllability to actuate 3D micro objects with actuators. For example, if the focus is on the object, the actuator could not be seen in the microscope. On the other hand, if the focus is on the actuator, the object could not be observed. In this sense, the is useful for the actuation in micro environments. However, one drawback on the all-in-focus image is that there is no information about the depth of objects. It is also important to reconstruct the micro 3D environments in real time to actuate the object with the micro virtual environments.This paper discusses the criteria of from to achieve the all-in-focus image and micro 3D reconstruction, simultaneously. Then, a micro VR camera system has been developed to achieve the micro VR environments with the from criteria, i.e. the combination of the all-in-focus image and the micro 3D reconstruction. This system is constructed by a dynamic focusing lens, which can change the focus in high frequency, and a smart vision system, which is capable in capturing and processing the image data in high speed with SIMD architecuture.

Amorphous silicon edge detectors for application to neural network image sensors

The initial steps in the biological visual signal processing are the extraction of the edge position and its orientation of an object image. These edge detection capabilities can be simulated by semiconductor edge detectors. In this letter, two types of such detectors, i.e., concentric and directional ones for detecting edge and its orientation, respectively, are successfully fabricated using an amorphous silicon hydrogen alloy. The measured performance of these edge detectors is similar to that of the biological ones. A three-dimensional image array, made by fabricating two-dimensional amorphous silicon edge detectors on top of the single-crystal Si very large-scale integrated circuit is proposed, which can be used as the preprocessor of a smart vision system.