Visual Data Processing Research Articles

Pedestrian recognition in multi-camera networks based on deep transfer learning and feature visualization

Abstract The extensive deployment of surveillance cameras in public places, such as subway stations and shopping malls, necessitates automated visual-data processing approaches to match pedestrians across non-overlapping multiple cameras. However, due to the insufficient number of labeled training samples in real surveillance scene, it is difficult to train an effective deep neural network for cross-camera pedestrian recognition. Moreover, the cross-camera variation in viewpoint, illumination, and background makes the task even more challenging. To address these issues, in this paper we propose to transfer the parameters of a pre-trained network to our target network and then update the parameters adaptively using training samples from the target domain. More importantly, we develop new network structures that are specially tailored for cross-camera pedestrian recognition task, and implement a simple yet effective multi-level feature fusion method that yield more discriminative and robust features for pedestrian recognition. Specifically, rather than conventionally perform classification on the single-level feature of the last feature layer, we instead utilize multi-level feature by associating feature visualization with multi-level feature fusion. As another contribution, we have published our codes and extracted features to facilitate further research. Extensive experiments are conducted on WARD, PRID and MARS datasets, we show that the proposed method consistently outperforms state-of-the-arts.

Hierarchical Temporal Memory Features with Memristor Logic Circuits for Pattern Recognition

Hierarchical temporal memory (HTM) is a machine learning algorithm inspired by the information processing mechanisms of the human neocortex and consists of a spatial pooler (SP) and temporal memory (TM). In this paper, we develop circuits and systems to achieve the optimized design of an HTM SP, an HTM TM, and a memristive analog pattern matcher for pattern recognition applications. The HTM SP realizes an optimized hardware design through the introduction of mean overlap calculations and by replacing the threshold determination in the inhibition stage with a weighted summation operator over the neighborhood of the pixel under consideration. HTM TM is based on discrete analog memristive memory arrays and a weight update procedure. The operation of the proposed system is demonstrated for a face recognition problem, using the standard AR, ORL, and Yale databases, and for speech recognition, using the TIMIT database, with achieved accuracies of 87.21% and approximately 90%, respectively, given an SNR of 10 dB. Visual data processing using binary HTM SP features requires less storage and processing memory than required by the traditional processing methods, with the area and power requirements for its implementation being 0.096 mm2 and 1756 mW, respectively. The design of the TM circuit for a single pixel requires 23.85 ${\mu}\text{m}^{2}$ of area and 442.26 ${\mu}\text{W}$ of power.

Augmented Reality Applications for Substation Management by Utilizing Standards-Compliant SCADA Communication

Most electrical substations are remotely monitored and controlled by using Supervisory Control and Data Acquisition (SCADA) applications. Current SCADA systems have been significantly enhanced by utilizing standardized communication protocols and the most prominent is the IEC 61850 international standard. These enhancements enable improvements in different domains of SCADA systems such as communication engineering, data management and visualization of automation process data in SCADA applications. Process data visualization is usually achieved through Human Machine Interface (HMI) screens in substation control centres. However, this visualization method sometimes makes supervision, control and maintenance procedures executed by engineers slow and error-prone because it separates equipment from its automation data. Augmented reality (AR) and mixed reality (MR) visualization techniques have matured enough to provide new possibilities of displaying relevant data wherever needed. This paper presents a novel methodology for visualizing process related SCADA data to enhance and facilitate human-centric activities in substations such as regular equipment maintenance. The proposed solution utilizes AR visualization techniques together with standards-based communication protocols used in substations. The developed proof-of-concept AR application that enables displaying SCADA data on the corresponding substation equipment with the help of AR markers demonstrates originality and benefits of the proposed visualization method. Additionally, the application enables displaying widgets and 3D models of substation equipment to make the visualization more user-friendly and intuitive. The visualized SCADA data needs to be refreshed considering soft real-time data delivery restrictions. Therefore, the proposed solution is thoroughly tested to demonstrate the applicability of proposed methodology in real substations.

Метод оцінки ефективності сегментації текстурних областей зображення

The article is devoted to the development of a method for evaluating the efficiency of segmentation of textual informative image areas, similar in color and structure, obtained as a result of aerial photography. Remote sensing of the Earth's surface provides unique opportunities for rapid acquisition of data with high spatial, spectral and temporal resolution. This fact determines the great information potential of such systems. Automated image processing of the Earth's surface allows us to solve scientific and applied problems in the field of cartography, research of the natural environment, oceanology, mineral exploration and development, agriculture and forestry and many other areas effectively. For example, the thematic classification of soil and vegetation cover based on aerospace survey data is a necessary stage of most tasks of landscape-ecological mapping.The article deals with models and methods for describing digital textural images. The current state of automated image processing in air monitoring systems is reviewed. The concept of a model of a textural image is defined. For the mathematical description of textures, two main approaches are used: structural and statistical. The first approach describes the basic areas of textures and the spatial relationships between them, which is well suited for describing artificial and regular textures. According to the second approach, the quantitative and statistical distribution of pixel intensity values in the image area is used as the texture description. Three main ways of image segmentation are considered in the article: threshold segmentation, segmentation by scaling up areas and segmentation by the boundaries detachment. Threshold segmentation consists in combining the image regions which are similar by characteristics into a small number of segments. If the brightness exceeds the threshold, then the region is assigned to one segment, and to another one in the opposite case. A review of existing methods for segmentation of textured image areas allowed us to formulate requirements for image processing algorithms for the operative processing of visual data in aerospace monitoring systems. . The input parameters are an aerial photograph, a priori information about the areas to be segmented, and meteorological and navigation-technical conditions for aerial photography. Output parameters are images with localized informative areas.

Path Control of Quadruped Robot through Convolutional Neural Networks

Abstract This paper deals with the application of the Convolutional Neural Networks (CNN) for path control of the autonomous quadruped robot. In this project, the quadruped robot detects the objects through the camera and visual data processing is performed via Python and OpenCV (Open Computer Vision). The OpenCV is a package that has many useful tools to analyze and process the images. The electronic components that are used in this project are Arduino microcontroller, Raspberry PI computer, Ultrasound detector and Pi camera. Regarding the software, in this project is developed by using the Python programming language, Numpy, Pyserial and Spyder libraries, which help to realize the autonomous movement of the quadruped robot.

Portable Brain-Computer Interface for the Intensive Care Unit Patient Communication Using Subject-Dependent SSVEP Identification.

A major predicament for Intensive Care Unit (ICU) patients is inconsistent and ineffective communication means. Patients rated most communication sessions as difficult and unsuccessful. This, in turn, can cause distress, unrecognized pain, anxiety, and fear. As such, we designed a portable BCI system for ICU communications (BCI4ICU) optimized to operate effectively in an ICU environment. The system utilizes a wearable EEG cap coupled with an Android app designed on a mobile device that serves as visual stimuli and data processing module. Furthermore, to overcome the challenges that BCI systems face today in real-world scenarios, we propose a novel subject-specific Gaussian Mixture Model- (GMM-) based training and adaptation algorithm. First, we incorporate subject-specific information in the training phase of the SSVEP identification model using GMM-based training and adaptation. We evaluate subject-specific models against other subjects. Subsequently, from the GMM discriminative scores, we generate the transformed vectors, which are passed to our predictive model. Finally, the adapted mixture mean scores of the subject-specific GMMs are utilized to generate the high-dimensional supervectors. Our experimental results demonstrate that the proposed system achieved 98.7% average identification accuracy, which is promising in order to provide effective and consistent communication for patients in the intensive care.

Medical Data Visualization Analysis and Processing Based on Machine Learning

Trying to provide a medical data visualization analysis tool, the machine learning methods are introduced to classify the malignant neoplasm of lung within the medical database MIMIC-III (Medical Information Mart for Intensive Care III, USA). The K-Nearest Neighbor (KNN), Support Vector Machine (SVM) and Random Forest (RF) are selected as the predictive tool. Based on the experimental result, the machine learning predictive tools are integrated into the medical data visualization analysis platform. The platform software can provide a flexible medical data visualization analysis tool for the doctors. The related practice indicates that visualization analysis result can be generated based on simple steps for the doctors to do some research work on the data accumulated in hospital, even they have not taken special data analysis training.

Efficient video collection association using geometry-aware Bag-of-Iconics representations

Recent years have witnessed the dramatic evolution in visual data volume and processing capabilities. For example, technical advances have enabled 3D modeling from large-scale crowdsourced photo collections. Compared to static image datasets, exploration and exploitation of Internet video collections are still largely unsolved. To address this challenge, we first propose to represent video contents using a histogram representation of iconic imagery attained from relevant visual datasets. We then develop a data-driven framework for a fully unsupervised extraction of such representations. Our novel Bag-of-Iconics (BoI) representation efficiently analyzes individual videos within a large-scale video collection. We demonstrate our proposed BoI representation with two novel applications: (1) finding video sequences connecting adjacent landmarks and aligning reconstructed 3D models and (2) retrieving geometrically relevant clips from video collections. Results on crowdsourced datasets illustrate the efficiency and effectiveness of our proposed Bag-of-Iconics representation.

Energy Consumption Models for Smart Camera Networks

Camera networks require heavy visual data processing and high-bandwidth communication. In this paper, we identify key factors underpinning the development of resource-aware algorithms and we propose a comprehensive energy consumption model for the resources employed by smart camera networks, which are composed of cameras that process data locally and collaborate with their neighbors. We account for the main parameters that influence consumption when sensing (frame size and frame rate), processing (dynamic frequency scaling and task load), and communication (output power and bandwidth) are considered. Next, we define an abstraction based on clock frequency and duty cycle that accounts for active, idle, and sleep operational states. We demonstrate the importance of the proposed model for a multicamera tracking task and show how one may significantly reduce consumption with only minor performance degradation when choosing to operate with an appropriately reduced hardware capacity. Moreover, we quantify the dependency on local computation resources and bandwidth availability. The proposed consumption model can be easily adjusted to account for new platforms, thus providing a valuable tool for the design of resource-aware algorithms and further research in resource-aware camera networks.

Multiresolution Volume Filtering in the Tensor Compressed Domain.

Signal processing and filter operations are important tools for visual data processing and analysis. Due to GPU memory and bandwidth limitations, it is challenging to apply complex filter operators to large-scale volume data interactively. We propose a novel and fast multiscale compression-domain volume filtering approach integrated into an interactive multiresolution volume visualization framework. In our approach, the raw volume data is decomposed offline into a compact hierarchical multiresolution tensor approximation model. We then demonstrate how convolution filter operators can effectively be applied in the compressed tensor approximation domain. To prevent aliasing due to multiresolution filtering, our solution (a) filters accurately at the full spatial volume resolution at a very low cost in the compressed domain, and (b) reconstructs and displays the filtered result at variable level-of-detail. The proposed system is scalable, allowing interactive display and filtering of large volume datasets that may exceed the available GPU memory. The desired filter kernel mask and size can be modified online, producing immediate visual results.

Object Detection and Recognition for Assistive Robots: Experimentation and Implementation

Technological advances are being made to assist humans in performing ordinary tasks in everyday settings. A key issue is the interaction with objects of varying size, shape, and degree of mobility. Autonomous assistive robots must be provided with the ability to process visual data in real time so that they can react adequately for quickly adapting to changes in the environment. Reliable object detection and recognition is usually a necessary early step to achieve this goal. In spite of significant research achievements, this issue still remains a challenge when real-life scenarios are considered. In this article, we present a vision system for assistive robots that is able to detect and recognize objects from a visual input in ordinary environments in real time. The system computes color, motion, and shape cues, combining them in a probabilistic manner to accurately achieve object detection and recognition, taking some inspiration from vision science. In addition, with the purpose of processing the input visual data in real time, a graphical processing unit (GPU) has been employed. The presented approach has been implemented and evaluated on a humanoid robot torso located at realistic scenarios. For further experimental validation, a public image repository for object recognition has been used, allowing a quantitative comparison with respect to other state-of-the-art techniques when realworld scenes are considered. Finally, a temporal analysis of the performance is provided with respect to image resolution and the number of target objects in the scene.

Study on Impact Acoustic-Visual Sensor-Based Sorting of ELV Plastic Materials.

This paper concentrates on a study of a novel multi-sensor aided method by using acoustic and visual sensors for detection, recognition and separation of End-of Life vehicles’ (ELVs) plastic materials, in order to optimize the recycling rate of automotive shredder residues (ASRs). Sensor-based sorting technologies have been utilized for material recycling for the last two decades. One of the problems still remaining results from black and dark dyed plastics which are very difficult to recognize using visual sensors. In this paper a new multi-sensor technology for black plastic recognition and sorting by using impact resonant acoustic emissions (AEs) and laser triangulation scanning was introduced. A pilot sorting system which consists of a 3-dimensional visual sensor and an acoustic sensor was also established; two kinds commonly used vehicle plastics, polypropylene (PP) and acrylonitrile-butadiene-styrene (ABS) and two kinds of modified vehicle plastics, polypropylene/ethylene-propylene-diene-monomer (PP-EPDM) and acrylonitrile-butadiene-styrene/polycarbonate (ABS-PC) were tested. In this study the geometrical features of tested plastic scraps were measured by the visual sensor, and their corresponding impact acoustic emission (AE) signals were acquired by the acoustic sensor. The signal processing and feature extraction of visual data as well as acoustic signals were realized by virtual instruments. Impact acoustic features were recognized by using FFT based power spectral density analysis. The results shows that the characteristics of the tested PP and ABS plastics were totally different, but similar to their respective modified materials. The probability of scrap material recognition rate, i.e., the theoretical sorting efficiency between PP and PP-EPDM, could reach about 50%, and between ABS and ABS-PC it could reach about 75% with diameters ranging from 14 mm to 23 mm, and with exclusion of abnormal impacts, the actual separation rates were 39.2% for PP, 41.4% for PP/EPDM scraps as well as 62.4% for ABS, and 70.8% for ABS/PC scraps. Within the diameter range of 8-13 mm, only 25% of PP and 27% of PP/EPDM scraps, as well as 43% of ABS, and 47% of ABS/PC scraps were finally separated. This research proposes a new approach for sensor-aided automatic recognition and sorting of black plastic materials, it is an effective method for ASR reduction and recycling.

Research on Online Mining Method of Real Time Soil Sensing Data in Farmland

With the wide application of various agriculture IOT sensor equipment, sensor networks produce ample data for observing dynamic soil properties in farmland. But data processing for analysis and visualization become increasingly difficult as data dimensionality increases. Meanwhile the devices sometimes lose data because of damage and power outages. In this study, exploratory data analysis (EDA) and spatiotemporal interpolation including time as a third dimension and spatial-temporal variogram are used to analyze the soil moisture and temperature data. The experimental results show that EDA can effectively reveal the implicit information of soil sensing data in multiple perspectives by a variety of visual ways. The leave-onestation- out cross-validation indicate that Kriging method based on spatial-temporal variogram has satisfied accuracy, which supplies an effective approach for interpolation and estimation of lost agriculture IOT sensing data. The results also have theoretic and practical value for regional soil management.

Big Data Processing for Full-Text Search and Visualization with Elasticsearch

In this paper, the task of using Big Data to identify specific individuals on the indirect grounds of their interaction with information resources is considered. Possible sources of Big Data and problems related to its processing are analyzed. Existing means of data clustering are considered. Available software for full-text search and data visualization is analyzed, and a system based on Elasticsearch engine and MapReduce model is proposed for the solution of user verification problem.

Knowledge extraction from a nitrification denitrification wastewater treatment plant using SOM-NG algorithm

ABSTRACTSOM-NG is a hybrid algorithm that is able to carry out visualization of process data, nonlinear function approximation, classification and clustering. The supervised version of SOM-NG produces a new type of 2D lattices called gradient planes which are useful to determine the dynamics of a target variable according to the remaining training variables. In this way, it is an interesting tool for data mining in order to extract knowledge from databases for nonlinear systems. The main objective of this work is to analyze data from an industrial wastewater treatment plant using SOM-NG algorithm in order to investigate relationships between the process variables. The data used proceeds from a biological wastewater treatment plant. This plant is based on an activated sludge treatment including nitrification and denitrification processes. A direct relation between the nitrification efficiency and the operating temperature was found, and also between the ammonia loading rate and the nitrification denitrification efficiency.

Visual Analytics of Complex Genomics Data to Guide Effective Treatment Decisions

In cancer biology, genomics represents a big data problem that needs accurate visual data processing and analytics. The human genome is very complex with thousands of genes that contain the information about the individual patients and the biological mechanisms of their disease. Therefore, when building a framework for personalised treatment, the complexity of the genome must be captured in meaningful and actionable ways. This paper presents a novel visual analytics framework that enables effective analysis of large and complex genomics data. By providing interactive visualisations from the overview of the entire patient cohort to the detail view of individual genes, our work potentially guides effective treatment decisions for childhood cancer patients. The framework consists of multiple components enabling the complete analytics supporting personalised medicines, including similarity space construction, automated analysis, visualisation, gene-to-gene comparison and user-centric interaction and exploration based on feature selection. In addition to the traditional way to visualise data, we utilise the Unity3D platform for developing a smooth and interactive visual presentation of the information. This aims to provide better rendering, image quality, ergonomics and user experience to non-specialists or young users who are familiar with 3D gaming environments and interfaces. We illustrate the effectiveness of our approach through case studies with datasets from childhood cancers, B-cell Acute Lymphoblastic Leukaemia (ALL) and Rhabdomyosarcoma (RMS) patients, on how to guide the effective treatment decision in the cohort.

Modeling and volume rendering approach of underwater three-dimensional acoustic energy field

The underwater acoustics is primary and most effective method for underwater object detection and the complex underwater acoustics battlefield environment can be visually described by the three-dimensional (3D) energy field. Through solving the 3D propagation models, the traditional underwater acoustics volume data can be obtained, but it is large amount of calculation. In this paper, a novel modeling approach, which transforms two-dimensional (2D) wave equation into 2D space and optimizes energy loss propagation model, is proposed. In this way, the information for the obtained volume data will not be lost too much. At the same time, it can meet the requirements of data processing for the real-time visualization. In the process of volume rendering, 3D texture mapping methods is used. The experimental results are evaluated on data size and frame rate, showing that our approach outperforms other approaches and the approach can achieve better results in real time and visual effects.

Augmented reality in neurosurgery: a systematic review.

Neuronavigation has become an essential neurosurgical tool in pursuing minimal invasiveness and maximal safety, even though it has several technical limitations. Augmented reality (AR) neuronavigation is a significant advance, providing a real-time updated 3D virtual model of anatomical details, overlaid on the real surgical field. Currently, only a few AR systems have been tested in a clinical setting. The aim is to review such devices. We performed a PubMed search of reports restricted to human studies of in vivo applications of AR in any neurosurgical procedure using the search terms "Augmented reality" and "Neurosurgery." Eligibility assessment was performed independently by two reviewers in an unblinded standardized manner. The systems were qualitatively evaluated on the basis of the following: neurosurgical subspecialty of application, pathology of treated lesions and lesion locations, real data source, virtual data source, tracking modality, registration technique, visualization processing, display type, and perception location. Eighteen studies were included during the period 1996 to September 30, 2015. The AR systems were grouped by the real data source: microscope (8), hand- or head-held cameras (4), direct patient view (2), endoscope (1), and X-ray fluoroscopy (1) head-mounted display (1). A total of 195 lesions were treated: 75 (38.46%) were neoplastic, 77 (39.48%) neurovascular, and 1 (0.51%) hydrocephalus, and 42 (21.53%) were undetermined. Current literature confirms that AR is a reliable and versatile tool when performing minimally invasive approaches in a wide range of neurosurgical diseases, although prospective randomized studies are not yet available and technical improvements are needed.

Resource Saving Approach of Visual Tracking Fiducial Marker Recognition for Unmanned Aerial Vehicle

Unmanned aerial vehicle (UAV) tracking fiducial marker is a challenging problem, because of camera system vibration, which causes visible frame-to-frame jitter in the airborne videos and unclear marker vision. Multirotors have very limited weight carrying, controller, and battery power resources. While obtaining and processing motion blurred images, which have no useful information, requires much more image processing subsystem resources. The paper presents blurry image frame elimination based approach of UAV resource saving fiducial marker visual tracking. The proposed approach integrates accelerometer and visual data processing algorithms to predict image blur and skip blurred frames. Experiments have been performed to verify the validity of the proposed approach.

Occurrence and species diversity of delphinids off-Lagos shore, Nigeria

Little has been documented about delphinid communities in the Nigerian coastal waters. This paper gives baseline information about the delphinid community sighted off-shore Lagos, Nigeria. A dedicated survey of the living resources of Nigerian coastal waters (6°08’N and 2°42’W to 6° 13’N and 3° 27’W ) took place from the Nigeria/Benin boarder along Badagry area to the western part of Ondo State between 17th march and 6th June 2009 and included systematic visual searching of marine mammals. A total effective effort of 264 hours was logged. The boat moved along designated transect lines at a cruising speed of 9.2 km/h. Sighting rate for delphinids was obtained by processing visual data obtained at Beaufort wind scale of 2.5 to 4 nautical miles in visibility, using Distance 2.2 software. Species were identified through Photo-identification method. Fifteen (15) schools of 746 individual’s sightings were made. Four (4) schools of 25 individuals representing 3% of total sightings were classified as “Unidentified”, while 11 schools of 721 individuals (97%) were identified and classified into five (5) species. The pantropical spotted dolphin ( Stenella attenuate ) with two (2) schools of 150 individuals, ( Stenella frontalis ) with one (1) school of 54 individuals, bottlenose dolphin ( Tursiops trucantus ) with one (1) school of 32 individuals, Common dolphin ( Delphinus delphis ) with four (4) schools of 452 individuals and (3) schools of Atlantic hump-backed dolphin ( Sousa teuszii ) of 32 individuals. Sighting rate of 5.7 x 10 -2 school/hr was estimated for schools while the sighting rate for individuals was estimated at 2.83 individual/hr. The results suggested that the Lagos coast is rich in delphinid biodiversity, which needs to be quantified more accurately during further research. Keywords : Sightings, Oceanic dolphins, abundance, species richness, behaviour, Nigeria.