Automatic Fusion Research Articles

Fusion of satellite, aircraft, and UAV data for automatic disaster damage assessment

ABSTRACTPost-disaster damage assessment is urgent for rapid save and rescue missions. Satellite and airborne imagery platforms take vertical images, in which only buildings’ roofs are observable. Unmanned aerial vehicles (UAVs) provide an alternative platform that can obtain oblique images from a scene but suffer from occlusion (hiding from view) and a smaller field of view. As imagery from each platform has several pros and cons, a fusion framework is required. A new framework is proposed in this article, based on building environment and image processing techniques to utilize the advantages of different platforms while avoiding the disadvantages. In this approach, a facade is detected in UAV oblique images by fusing a geometrical transformation and environmental information from vertical and oblique images. Fusion of roof and facade features is used to improve damage scale estimation. Damage is estimated automatically for both roof-only and facade-plus-roof images that are sourced from satellite and fused data respectively. The proposed approach is applied to several different datasets including the Haiti earthquake of 2010, Hurricane Irene of 2011, Hurricane Sandy of 2012 and the Illinois tornadoes of 2015. The proposed automatic fusion framework led to enhanced damage scale estimation in comparison with the roof-only damage assessment in these cases.

Full-Automatic Image Fusion of Real-Time Ultrasound and MR/CT Images for Radiofrequency Ablation of Hepatic Malignancies

To compare the registration error, time required for the image fusion and clinical outcomes of manual and full-automatic image fusion of ultrasound (US) and MR/CT images for radiofrequency ablation (RFA) of HCC and colorecatl cancer liver metastasis (CRLM). 67 patients with HCC or CRLM were prospectively enrolled. Patients are randomly assigned to a) manual fusion group, or b) automatic fusion group. Two interventional radiologists performed RFA for HCCs or CRLM with manual or full-automatic image fusion of real-time US and MR/CT. Time for image registration (sec), number of point locks, registration error (mm) and technical success rate were compared between the manual group and the automatic group. 32 HCCs and one CRLM were treated using the manual image fusion and 34 HCCs were treated using the automatic image fusion. Mean sizes of HCCs were 1.60 cm for the manual fusion and 1.63 cm for the automatic fusion (p=0.903). Averages of time for image registration, number of point locks and the registration error were 149 sec, 3.6 and 6.0 mm for the manual group, and 147 sec, 3.4 and 5.6 mm for the automatic group, respectively, and there were no significant difference. Technical success rate was 97.0% (32/33) for the manual group and 97.1% (33/34) for the automatic group and there was no significant difference between each other. (p=1.000). The technical performance of the full-automatic fusion technique is comparable to that of manual fusion technique.

Surface imaging, laser positioning or volumetric imaging for breast cancer with nodal involvement treated by helical TomoTherapy.

A surface imaging system, Catalyst (C‐Rad), was compared with laser‐based positioning and daily mega voltage computed tomography (MVCT) setup for breast patients with nodal involvement treated by helical TomoTherapy. Catalyst‐based positioning performed better than laser‐based positioning. The respective modalities resulted in a standard deviation (SD), 68% confidence interval (CI) of positioning of left–right, craniocaudal, anterior–posterior, roll: 2.4 mm, 2.7 mm, 2.4 mm, 0.9° for Catalyst positioning, and 6.1 mm, 3.8 mm, 4.9 mm, 1.1° for laser‐based positioning, respectively. MVCT‐based precision is a combination of the interoperator variability for MVCT fusion and the patient movement during the time it takes for MVCT and fusion. The MVCT fusion interoperator variability for breast patients was evaluated at one SD left–right, craniocaudal, ant–post, roll as: 1.4 mm, 1.8 mm, 1.3 mm, 1.0°.There was no statistically significant difference between the automatic MVCT registration result and the manual adjustment; the automatic fusion results were within the 95% CI of the mean result of 10 users, except for one specific case where the patient was positioned with large yaw. We found that users add variability to the roll correction as the automatic registration was more consistent.The patient position uncertainty confidence interval was evaluated as 1.9 mm, 2.2 mm, 1.6 mm, 0.9° after 4 min, and 2.3 mm, 2.8 mm, 2.2 mm, 1° after 10 min. The combination of this patient movement with MVCT fusion interoperator variability results in total standard deviations of patient position when treatment starts 4 or 10 min after initial positioning of, respectively: 2.3 mm, 2.8 mm, 2.0 mm, 1.3° and 2.7 mm, 3.3 mm, 2.6 mm, 1.4°.Surface based positioning arrives at the same precision when taking into account the time required for MVCT imaging and fusion. These results can be used on a patient‐per‐patient basis to decide which positioning system performs the best after the first 5 fractions and when daily MVCT can be omitted. Ideally, real‐time monitoring is required to reduce important intrafraction movement.PACS number(s): 87.53.Jw, 87.53.Kn, 87.56.Da, 87.63.L‐, 81.70.Tx

Manual versus automatic image fusion of real-time ultrasonography and MR/CT images for radiofrequency ablation of hepatic tumors: A preliminary report of a randomized prospective trial

Manual versus automatic image fusion of real-time ultrasonography and MR/CT images for radiofrequency ablation of hepatic tumors: A preliminary report of a randomized prospective trial

CO-REGISTRATION OF DSMs GENERATED BY UAV AND TERRESTRIAL LASER SCANNING SYSTEMS

An approach for the co-registration of Digital Surface Models (DSMs) derived from Unmanned Aerial Vehicles (UAVs) and Terrestrial Laser Scanners (TLS) is proposed. Specifically, a wavelet-based feature descriptor for matching surface keypoints on the 2.5D DSMs is developed. DSMs are useful in wide-scope of various applications such as 3D building modelling and reconstruction, cultural heritage, urban and environmental planning, aircraft navigation/path routing, accident and crime scene reconstruction, mining as well as, topographic map revision and change detection. For these listed applications, it is not uncommon that there will be a need for automatically aligning multi-temporal DSMs which may have been acquired from multiple sensors, with different specifications over a period of time, and may have various overlaps. Terrestrial laser scanners usually capture urban facades in an accurate manner; however this is not the case for building roof structures. On the other hand, vertical photography from UAVs can capture the roofs. Therefore, the automatic fusion of UAV and laser-scanning based DSMs is addressed here as it serves various geospatial applications.

CO-REGISTRATION OF DSMs GENERATED BY UAV AND TERRESTRIAL LASER SCANNING SYSTEMS

Abstract. An approach for the co-registration of Digital Surface Models (DSMs) derived from Unmanned Aerial Vehicles (UAVs) and Terrestrial Laser Scanners (TLS) is proposed. Specifically, a wavelet-based feature descriptor for matching surface keypoints on the 2.5D DSMs is developed. DSMs are useful in wide-scope of various applications such as 3D building modelling and reconstruction, cultural heritage, urban and environmental planning, aircraft navigation/path routing, accident and crime scene reconstruction, mining as well as, topographic map revision and change detection. For these listed applications, it is not uncommon that there will be a need for automatically aligning multi-temporal DSMs which may have been acquired from multiple sensors, with different specifications over a period of time, and may have various overlaps. Terrestrial laser scanners usually capture urban facades in an accurate manner; however this is not the case for building roof structures. On the other hand, vertical photography from UAVs can capture the roofs. Therefore, the automatic fusion of UAV and laser-scanning based DSMs is addressed here as it serves various geospatial applications.

SU‐G‐JeP4‐12: Real‐Time Organ Motion Monitoring Using Ultrasound and KV Fluoroscopy During Lung SBRT Delivery

Purpose:Real‐time ultrasound monitoring during SBRT is advantageous in understanding and identifying motion irregularities which may cause geometric misses. In this work, we propose to utilize real‐time ultrasound to track the diaphragm in conjunction with periodical kV fluoroscopy to monitor motion of tumor or landmarks during SBRT delivery.Methods:Transabdominal Ultrasound (TAUS) b‐mode images were collected from 10 healthy volunteers using the Clarity Autoscan System (Elekta). The autoscan transducer, which has a center frequency of 5 MHz, was utilized for the scans. The acquired images were contoured using the Clarity Automatic Fusion and Contouring workstation software. Monitoring sessions of 5 minute length were observed and recorded. The position correlation between tumor and diaphragm could be established with periodic kV fluoroscopy periodically acquired during treatment with Elekta XVI. We acquired data using a tissue mimicking ultrasound phantom with embedded spheres placed on a motion stand using ultrasound and kV Fluoroscopy. MIM software was utilized for image fusion. Correlation of diaphragm and target motion was also validated using 4D‐MRI and 4D‐CBCT.Results:The diaphragm was visualized as a hyperechoic region on the TAUS b‐mode images. Volunteer set‐up can be adjusted such that TAUS probe will not interfere with treatment beams. A segment of the diaphragm was contoured and selected as our tracking structure. Successful monitoring sessions of the diaphragm were recorded. For some volunteers, diaphragm motion over 2 times larger than the initial motion has been observed during tracking. For the phantom study, we were able to register the 2D kV Fluoroscopy with the US images for position comparison.Conclusion:We demonstrated the feasibility of tracking the diaphragm using real‐time ultrasound. Real‐time tracking can help in identifying such irregularities in the respiratory motion which is correlated to tumor motion. We also showed the feasibility of acquiring 2D KV Fluoroscopy and registering the images with Ultrasound.

Automatic fusion of visible and infrared images taken from different perspectives for diagnostics of power lines

Power transmission lines are ones of technical structures that are needed to be diagnosed regularly. Inspections very often are performed with use of various types of cameras, primarily vision and infrared (IR) ones. The author of this paper proposes an approach to automated fusion of visible and IR images that were taken from different perspectives. Such an operation enables automated merging and matching of such a pair of images as well as returning needed information about a potential failure presence and its location in a single image. The algorithm is based on Speeded Up Robust Features detection and is scale- and rotate-invariant. The method has brought satisfying results since the vast majority of considered cases of pairs of images were merged properly and a failure, visible as a hot spot, was additionally highlighted and spatially localised in resulting images, which hence facilitates interpretation of results of inspection with respect to analysis of the separate images. This approach enables automation of detection, localisation and visualisation of a failure or abnormality in operation of power lines. It can be applied for the purposes of diagnostics of power transmission lines as well as other industrial applications, e.g. inspections of buildings, bridges, pipelines, etc.

Comparison study of 3DUS-US and CT-US image fusion technology based on electromagnetic positioning

Objective To compare the accuracy, convenience and repeatability of 3DUS-US and CT-US image fusion technology based on electromagnetic positioning. Methods A tissue-mimicking phantom was established and used to obtain ultrasound or CT volume images. Two different operators performed 3DUS-US and CT-US image fusion and repeated 10 times, respectively. The success rate, the registration error distance and fusion time of two techniques were recorded and compared between the different operators. Results The ultrasound and CT images of the phantom and its stability could meet the demands of this experiment. 3DUS-US and CT-US image fusions were successful. The registration error distance of 3DUS-US image fusion was (1.70±0.42)mm and fusion time was (76.00±9.99)s, they were obviously superior to CT-US (P=0.014, P<0.001). There were no significant differences between the two operators in the registration error distance and fusion time of 3DUS-US (P=0.508, P=0.517). But the registration error distance of CT-US image fusion in experienced operator was lower than the junior (P=0.009), and fusion time had not statistical difference between the two operators (P=0.234). Conclusions The technique of 3DUS-US automatic image fusion based on electromagnetic positioning has advantages of convenience and no experiential dependence comparing with CT-US in the phantom experiments, so it is worthy of being widely popularized in clinical application. Key words: Ultrasonography; Image fusion; Tomography, X-ray computed; Electromagnetic positioning; Phantom

Experimental study of 3DUS-US single-modality automatic image fusion technology based on electromagnetic positioning

Objective To evaluate the feasibility and accuracy of three-dimensional ultrasound-ultrasound (3DUS-US) automatic image fusion technology based on electromagnetic positioning. Methods The experimental phantom was constructed to acquire ultrasonic images by agarose gel and additives, which was used as the experimental object. 3DUS imaging with free-hand and real-time ultrasound imaging automatic registrations were performed based on electromagnetic positioning. To investigate the effect of 3DUS-US image fusion under different scanning speed by free-hand, the fast and slow groups were designed. In addition, a junior operator and a senior operator performed 3DUS-US automatic registrations, and recorded the operating time and registration error, respectively. The repeatability between two operators was analyzed. Results The macroscopic appearance, stability and ultrasonic image of the phantom met the demand of this research. The success rate of 3DUS-US automatic image fusion technology was 100% (40/40). The slow group's registration error distance was (1.44±0.64)mm, which was obviously lower than the fast group's (2.56±0.53)mm, the difference was statistically significant (P<0.001). There were no statistically differences of the registration error and operating time between the two operators (P=0.508, P=0.517). Conclusions The technology of 3DUS-US automatic registration based on electromagnetic positioning was feasible and accurate, which is worth applying into the clinical treatment. Key words: Ultrasonography, three-dimensional; Image fusion; Electromagnetic positioning; Phantom

Automatic fusion of lateral cephalograms and digital volume tomography data-perspective for combining two modalities in the future.

This article investigates the combination of three-dimensional (3D) digital volume tomography data with two-dimensional (2D) cephalograms in dentomaxillofacial imaging. An automatic hierarchical method to adjust the geometrical relations of these two modalities is presented. The approach is tested on phantom and patient case data, where the feasibility, usability and potential possibilities of the presented innovative method are highlighted. Digitally reconstructed radiographs are computed by casting rays through the 3D volume to get a 2D projection of the volume to produce realistic simulated cephalograms. Different similarity measures are considered based on variations of statistical and deterministic optimization procedures. Stability, precision and accuracy of the method are investigated. The presented algorithm demonstrates a reasonable solution of the corresponding 2D/3D registration problem. Exemplary results from phantom and patient case data are presented. Tooth movement could be determined, in contrast to the 2D lateral cephalogram, separated for each side in all three spatial directions. Achieved results are highlighted from a clinical point of view and demonstrate the clinical benefit in daily praxis.

EP-1497: 4D-CBCT frameless image guidance in lung SBRT: Comparison between 4D automatic fusion algorithm and manual fusion

EP-1497: 4D-CBCT frameless image guidance in lung SBRT: Comparison between 4D automatic fusion algorithm and manual fusion

Design and development of an online geoinformation service delivery of geospatial models in the United Kingdom

This paper presents the latest design and realisation of a low-cost innovative online secure geoprocessing service, also known as web processing service, which undertakes fully automated real-time, online, on-demand precise geo-referenced spatial and attribute data extraction, transfer and fusion. The service carries out subsequently immediate generation of quasi-realistic geospatial population models for end users in the United Kingdom’s academic community. The paper presents realised system architecture, design and operational mechanisms, processes and workflow for automatic precise online service invocation, extraction of geospatial data and attributes data from online data-intensive digital repositories and services and automatic data fusion of assured integrity and quality. As an integral part, these include following-on automatic high performance geoprocessing for generating geospatial models and service delivery over the Internet. It explains how a set of specially designed and coded geoprocessors and associated geo-referencing coding system are produced to perform coherently chained actions in a novel workflow to act in unison to meet the end users demand. It is believed that such types of automated geodata processing services can be used for on-demand processing of an ever increasing amount of environmental data.

Diffuse hepatic diseases affect on three-dimensional ultrasound single-modality automatic images fusion

Objective To explore how diffuse hepatic diseases affect on the success rate and registration accuracy of three-dimensional ultrasound (3DUS) automatic images fusion technology. Methods 3DUS volumetric images of right hepatic anterior segment were acquired from 30 volunteers including healthy (10 cases), fatty liver (10 cases) and cirrhosis (10 cases). 3DUS automatic registrations were performed based on hepatic vessel in each pair of images.Meanwhile, the picture qualities were scored in all 3DUS volumetric images. How diffuse hepatic disease and picture quality affect on 3DUS single-modality automatic images fusion technology were analyzed. Results Twenty-nine pairs of 3DUS volumetric images were performed automatic registrations, the success rate was about 89.7%(26/29) and registration error distance was 1.31 mm(1.10~1.99 mm). Comparison to healthy controls, the success rate and accuracy of 3DUS image fusion decreased in fatty liver and cirrhosis, but there were no statistical differences (P=0.507, P=0.167). Picture quality could affect the success rate, and there was positive correlation between them (r=0.536, P=0.022). Conclusions 3DUS single-modality automatic images fusion technology has a good applicability in diffuse hepatic diseases, but picture quality can affect the success rate of this new technology. Key words: Ultrasonography, three-dimensional; Fatty liver; Liver cirrhosis; Image fusion

Automatic Fusion of Hyperspectral Images and Laser Scans Using Feature Points

Automatic fusion of different kinds of image datasets is so intractable with diverse imaging principle. This paper presents a novel method for automatic fusion of two different images: 2D hyperspectral images acquired with a hyperspectral camera and 3D laser scans obtained with a laser scanner, without any other sensor. Only a few corresponding feature points are used, which are automatically extracted from a scene viewed by the two sensors. Extraction method of feature points relies on SURF algorithm and camera model, which can convert a 3D laser scan into a 2D laser image with the intensity of the pixels defined by the attributes in the laser scan. Moreover, Collinearity Equation and Direct Linear Transformation are used to create the initial corresponding relationship of the two images. Adjustment is also used to create corrected values to eliminate errors. The experimental result shows that this method is successfully validated with images collected by a hyperspectral camera and a laser scanner.

Demos

In today's dynamic network environments like smart cities, information evolves in distributed resources and in different forms or modalities. An agent or a system that wants to be actively part of this evolution has to be interactive, adaptive, autonomous and intelligent. The paper presents a new version of a distributed model-framework (called Demos) based on autonomous, intelligent agents with anticipatory responses. The first version of the system was proposed by , and the new features (LG Graphs, SPNs and NNs) were embedded in the version presented here. The Demos model is mainly based on the development of an adaptive distributed knowledge base system. Knowledge is represented in a form of frames with internal stochastic Petri-net graph for local representations (KR). A major advantage for the Demos model having a distributed (possibly across the net) is the adaptive knowledge base. Here the authors present the design of an adaptive knowledge model and its challenge, which lays principally on how to learn new knowledge by synthesising SPN forms, and how to develop anticipatory responses at an agent's site. The development of the Demos prototype has a great range of applicability, such as in autonomous negotiating teams, autonomous distributed units for energy efficient distribution, autonomous multiple mobile robots space exploration and maps generation, autonomous intelligent information agents (WWW), automatic information synthesis and fusion, etc. Here the authors have used this model for management of the electric power on a grid of a smart city.

Automatic histogram-based fuzzy C-means clustering for remote sensing imagery

Fuzzy C-means (FCM) clustering has been widely used in analyzing and understanding remote sensing images. However, the conventional FCM algorithm is sensitive to initialization, and it requires estimations from expert users to determine the number of clusters. To overcome the limitations of the FCM algorithm, an automatic histogram-based fuzzy C-means (AHFCM) algorithm is presented in this paper. Our proposed algorithm has two primary steps: 1 – clustering each band of a multispectral image by calculating the slope for each point of the histogram, in two directions, and executing the FCM clustering algorithm based on specific rules, and 2 – automatic fusion of labeled images is used to initialize and determine the number of clusters in the FCM algorithm for automatic multispectral image clustering. The performance of our proposed algorithm is first tested on clustering a very high resolution aerial image for various numbers of clusters and, next, on clustering two very high resolution aerial images, a high resolution Worldview2 satellite image, a Landsat8 satellite image and an EO-1 hyperspectral image, for a constant number of clusters. The superiority of the new method is demonstrated by comparing it with the well-known methods of FCM, K-means, fast global FCM (FGFCM) and kernelized fast global FCM (KFGFCM) clustering algorithms, both quantitatively by calculating the DB, XB and SC indices and qualitatively by visualizing the cluster results.

Mechanical and Electrical Integration in the Application of the Excavator

Mechanical and electrical integration, also known as mechanical, electronics is an interdisciplinary comprehensive high technology, is made up of microelectronics technology, computer technology, information technology, automatic control technology, mechanical technology, hydraulic technology and other technology cross mutual fusion and become an independent discipline. This article mainly discusses the mechanical and electrical integration in the application of all kinds of excavator.

Image guidance procedures in radiotherapy for prostate cancer and the influence of body mass index

AbstractPurposeTo investigate possible optimisation of the image guidance procedure for the prostate cancer patients with respect to imaging frequency and patient body mass index (BMI).MethodsThe 6,085 setup correction shifts and BMI for 216 prostate cancer patients treated on tomotherapy units in two centres were analysed. Margins needed to account for inter-fraction target motion with daily only automatic correction and with automatic and manual corrections during one, three or five first fractions as a reference for further treatment without imaging were calculated.ResultsThe planning target volume margin calculated for the daily automatic correction only scheme was significantly lower than the margins calculated for the image guidance limited to a few initial fractions. Manual corrections after automatic fusion were more important for patients with higher BMI. On average, the patients with normal BMI had manual correction shift of 0·7 mm in anterioposterior direction, while overweight and obese patients required, correspondingly, the shifts of 1·3 and 1·4 mm.ConclusionOverweight and obese patients require daily imaging with time saving available by performing automatic kV/MV computed tomography registration only. The patients with normal BMI may be treated with imaging guidance during a few initial treatment fractions.

Automated proton magnetic resonance spectroscopy imaging guided frameless stereotactic biopsy of intracranial lesions

To evaluate the feasibility, reliability and accuracy of the automated magnetic resonance spectroscopy ((1)H-MRS) guided frameless brain biopsy with intraoperative magnetic resonance imaging (iMRI). Between July 2011 and July 2013, a consecutive series of 93 patients were prospectively enrolled. All the patients had intracranial lesions which need biopsy to confirm the diagnosis. Among them, 48 patients were male, 45 patients were female. Their age range from 7 years to 76 years, the median age was 47 years. All patients underwent MRS examination. With MRS automatic fusion technique, the metabolic images were integrated into a standard navigation system (Vario Guide) to guide frameless biopsy. High-field iMRI (1.5 T) was used for target inspection, brain shift correction, and intra-operative exclusion of intra-cerebral hemorrhage and other complications. For all the 93 patients, (1)H-MRS based metabolic images could be automatically integrated into a standard navigation system and average fusion procedure could be taken 5 minutes 6 seconds. For (1)H-MRS guided stereotactic biopsy of intracranial lesions, the diagnosis yield rate was 94.6% (88/93). Four cases did not get a clear pathological diagnosis, while 1 case did not match the pathological diagnosis result which obtained by following craniotomy. Technical related complication rate was 2.2% (2 cases, intra-cerebral hemorrhage), which were intra-operatively depicted with iMRI, and managed properly. Among them, 1 case with small volume (5 ml) intracerebral hematoma fully recovered 10 days after surgery without second surgical intervention. One case with large volume intracerebral hematoma (32 ml) was depicted with iMRI, followed by craniotomy and hematoma evacuation in the same session. This case had no new or worsened neurologic deficit post-operatively. (1)H-MRS based metabolic imaging can be automatically integrated into a standard navigation system and used for frameless brain biopsy. The target can be selected according to the metabolic status of the lesion. Hence, the target can be more accurate. And the pathological diagnosis yield rate is higher. With iMRI, the method is safe, and has high clinical efficacy.