Full HD Images Research Articles

FaSS-MVS: Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-Borne Monocular Imagery.

With FaSS-MVS, we present a fast, surface-aware semi-global optimization approach for multi-view stereo that allows for rapid depth and normal map estimation from monocular aerial video data captured by unmanned aerial vehicles (UAVs). The data estimated by FaSS-MVS, in turn, facilitate online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated as the image data are acquired or being received. FaSS-MVS is composed of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing efficient processing of large scene depths, such as those inherent in oblique images acquired by UAVs flying at low altitudes. The actual depth estimation uses a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of Semi-Global Matching (SGM). Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and computing the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study, we show that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP. At the same time, however, the average runtime of FaSS-MVS for estimating a single depth and normal map is less than 14% of that of COLMAP, allowing us to perform online and incremental processing of full HD images at 1-2 Hz.

Shortening replacement by hydroxypropyl methylcellulose-based oleogels obtained by different indirect approaches. Texture and sensory properties of baked puff pastry

Shortening in puff pastry products contains high levels of saturated and even trans-fatty acids that may be hazardous to human health. One efficient way to reduce such unhealthy fat in food is to employ gelators to structure liquid oils for the production of oleogels, thereby replacing shortening. In this paper, two types of HPMC (Hydroxypropyl Methyl Cellulose) oleogels (emulsion template and foam template) were used to replace shortening in puff pastry. Full-HD images, TPA (Texture Profile Analysis) and penetration tests, as well as TDS (Time-Dominant Sensation) sensory analysis, were used to assess the products' cross-sectional morphology, texture, and sensory attributes. The appearance of baked pastry made with 50/50 (shortening/oleogel) was similar to the control (100% shortening). In contrast, the crumb texture of baked pastry prepared with 100% oleogel was more compact and less airy than control. All samples showed similar firmness. TPA test showed that puff pastry made with 100% foam-based oleogel did not substantially differ to the control group, while samples prepared with emulsion-based oleogels were significantly harder and chewier than the control. Regarding sensory analysis, all puff pastries were mainly perceived as "crunchy" during chewing. The increase in oleogel content resulted in denser pastry, but the 50/50 foam template samples were "easier to chew". Attribute "compact" also became one of the main perceptions when the oleogel content increased to 100%, which would be related to the dense structure observed in the crumb morphology. This study confirms the feasibility of HPMC oleogels as healthy fat source in puff pastry.

FPGA Design of Real Time Hardware for Face Detection

This paper proposes the hardware architecture of face detection FPGA hardware system using the AdaBoost algorithm. The proposed structure of face detection hardware system is possible to work in 30 frames per second and in real time. And the AdaBoost algorithm is adopted to learn and generate the characteristics of the face data by MATLAB, and finally detected the face using this data. This paper describes the face detection hardware structure composed of image scaler, integral image extraction, face comparing, memory interface, data grouper and detected result display. The proposed circuit is so designed to process one point in one cycle that the proposed design can process full HD (1920x1080) image at 70MHz, which is approximate 2316087 x 30 cycle.

Using a two-stage convolutional neural network to rapidly identify tiny herbivorous beetles in the field

Recently, deep convolutional neural networks (CNN) have been adopted to help non-experts identify insect species from field images. However, the application of these methods on the rapid identification of tiny congeneric species moving across heterogeneous background remains difficult. To improve rapid and automatic identification in the field, we customized an existing CNN-based method for a field video involving two Phyllotreta beetles. We first performed data augmentation using transformations, syntheses, and random erasing of the original images. We then proposed a two-stage method for the detection and identification of small insects based on CNN, where YOLOv4 and EfficientNet were used as a detector and a classifier, respectively. Evaluation of the model revealed that one-step object detection by YOLOv4 alone was not precise (Precision=0.55) when classifying two species of flea beetles and background objects. In contrast, the two-step CNNs improved the precision (Precision=0.89) with moderate accuracy (F-measure=0.55) and acceptable speed (ca. 5 frames per second for full HD images) of detection and identification of insect species in the field. Although real-time identification of tiny insects remains a challenge in the field, our method aids in improving small object detection on a heterogeneous background.

Video Abstracts

Purpose/Background: It is well recognised that patients following long course chemoradiotherapy can have a complete clinical response. This offers the possibility for organ preservation. However these patients require intense surveillance which commonly involves frequent MRI scans and flexible sigmoidoscopy. LumenEye is a state of the art endoscopic kit, which allows a convenient, comfortable and efficient way of visualising the ano-rectal area in OPD with high quality images avoiding the need for a formal endoscopic procedure. Methods/Interventions: The device includes (a) LumenEye X1 endoscope (b) Tablet display with full HD screen (c) Intuitive CHiP software and (d) Portable case with workstation. The kit displays full HD images and can capture videos. Results/Outcomes: We have used the LumenEye X1 endoscope to survey patients following a complete clinical response who have opted for organ preservation. It avoids the need for an endoscopy and detailed images and videos can be shared with specialist centres. This has been of particular benefit during the Covid 19 pandemic where there has been a limited endoscopy resource. The technology also has other applications such as the intraoperative assessment of left sided colonic anastomosis. Conclusion/Discussion: LumenEye X1 endoscope offers an alternative to the use of endoscopy in the surveillance of patients with a complete clinical response. It also has the potential to free up valuable endoscopy resources.

VRU Pose-SSD: Multiperson Pose Estimation For Automated Driving

We present a fast and efficient approach for joint person detection and pose estimation optimized for automated driving (AD) in urban scenarios. We use a multitask weight sharing architecture to jointly train detection and pose estimation. This modular architecture allows us to accommodate different downstream tasks in the future. By systematic large-scale experiments on the Tsinghua-Daimler Urban Pose Dataset (TDUP), we obtain multiple models with varying accuracy-speed trade-offs. We then quantize and optimize our network for deployment and present a detailed analysis of the efficacy of the algorithm. We introduce a two-stage evaluation strategy, which is more suitable for AD and achieve a significant performance improvement in comparison to state-of-the-art approaches. Our optimized model runs at 52~fps on full HD images and still reaches a competitive performance of 32.25~LAMR. We are confident that our work serves as an enabler to tackle higher-level tasks like VRU intention estimation and gesture recognition, which rely on stable pose estimates and will play a crucial role in future AD systems.

A High-Throughput Depth Estimation Processor for Accurate Semiglobal Stereo Matching Using Pipelined Inter-Pixel Aggregation

Semiglobal matching is an accurate stereo depth estimation algorithm, whereas implementing the high-throughput architecture has been challenging due to the inherent recursion on inter-pixel cost aggregation. Especially, the computation on horizontal scan pass is the critical path causing the throughput bottleneck. In this paper, we propose a new cluster-wise cost aggregation algorithm and its optimized architecture that enables to pipeline the inter-pixel aggregation and parallelize the scanline-level disparity computation. The proposed approach is performed not on every pixel but on each group of pixels, which significantly alleviates the timing constraint for the recursion. The disparity values at shifted multiple pixel positions are concurrently computed within a single clock period. We also propose the memory reduction scheme selecting a tiny number of informative values, which achieves 96% memory reduction compared to the straightforward approach storing overall values. The system-on-chip-based tiled processing scheme is employed, which allows the implementation without an external memory. The proposed architecture computes a depth map with 128 disparity levels at 103 frames per second on a full HD image on the Zynq ultrascale+ MPSoC platform, thus providing 2.6 times faster performance with a comparable accuracy compared to the state-of-the-art 8-path semiglobal matching implementation.

The Application of the Model of High-Speed Pixel Clustering in Problems of Preprocessing of the Images of the Remote Sensing of the Earth

The purpose of the research is to use the modified Ward’s method in high-speed processing of full-HD images of the remote sensing of the Earth. The classical Ward’s method is modified by dividing the computational process into three successive stages. The first stage quickly builds a coarse hierarchy of approximations. The second stage performs a quality improvement of the specified partition for a fixed number of colors (clusters). The third stage is the clustering of the superpixels using the Ward’s method. The software-algorithmic toolkit consists of four operations on clusters of pixels and image segments: merge operation joins together two clusters; divide operation reversibly disjoins the selected cluster into two; split operation extracts the part of the cluster into individual cluster; correct operation reclassifies pixels by extracting from one cluster and inserting into another cluster. The quality is assessed by the total squared error. The quality improvement is provided by iterative execution of a combination of merge and divide operations of pixel clusters, in particular image segments. One of the clusters (segments) is divided in two and a pair of other mismatched with it is combined into one according to the criterion of the minimum increment of the total squared error. The proposed modified Ward’s method is appropriate in processing of fullHD images of the remote sensing of the Earth. The results of processing in pure segmentation and clustering modes are compared. The proposed pixel clustering model is appropriate in high-speed processing of the fullHD images. The pixel clustering in comparison with image segmentation allows to define in more detail both the contours of objects of interest and their internal structure

Tillage boundary detection based on RGB imagery classification for an autonomous tractor

In this study, a deep learning-based tillage boundary detection method for autonomous tillage by a tractor was developed, which consisted of image cropping, object classification, area segmentation, and boundary detection methods. Full HD (1920 × 1080) images were obtained using a RGB camera installed on the hood of a tractor and were cropped to 112 × 112 size images to generate a dataset for training the classification model. The classification model was constructed based on convolutional neural networks, and the path boundary was detected using a probability map, which was generated by the integration of softmax outputs. The results show that the F1-score of the classification was approximately 0.91, and it had a similar performance as the deep learning-based classification task in the agriculture field. The path boundary was determined with edge detection and the Hough transform, and it was compared to the actual path boundary. The average lateral error was approximately 11.4 cm, and the average angle error was approximately 8.9o. The proposed technique can perform as well as other approaches; however, it only needs low cost memory to execute the process unlike other deep learning-based approaches. It is possible that an autonomous farm robot can be easily developed with this proposed technique using a simple hardware configuration.

A Flexible and Efficient Real-Time ORB-Based Full-HD Image Feature Extraction Accelerator

This article proposes a hardware accelerator for image feature extraction based on the oriented features from accelerated segment test (FAST) and rotated binary robust independent elementary features (ORB) algorithm. The architecture adopts a hybrid workflow to process three scales in parallel as an octave, and multiple octaves are processed in serial by time-sharing the DRAM. In order to support any image resolution, a block-wise dataflow is used on the serial-parallel architecture. The overlapped data between two blocks is reused. As a result, the on-chip memory is limited to 1.47 Mb, and the DRAM bandwidth is compressed by 33%. Besides, 3×3 nonmaximum suppression with heap sorting is applied to balance keypoint distribution in 2-D. It improves the valid keypoint match ratio by 7.09%. In addition, parallel processing in keypoint detection increases 2× throughout. Moreover, approximate computing and superscalar processing reduce the timing cost in orientation estimation and descriptor generation by 92.78% and 67% respectively. Compared to the non-optimized baseline architecture, the proposed architecture saves the total timing cost by 82.4%. The accelerator is implemented in a Xilinx MPSoC, and it achieves 108 fps on full-HD images at 200 MHz under 873 mW.

Hardware Design of a Context-Preserving Filter-Reorganized CNN for Super-Resolution

This paper presents a hardware design of a CNN for single image super-resolution (SISR) processing. Very deep convolutional network for image super-resolution (VDSR) is a promising algorithm for SISR generating high-quality images but it is too complex to be implemented in hardware for commercial products. The paper presents a novel hardware design that is applicable to commercial products with low hardware resource usage while avoiding a significant degradation of image quality. The proposed design employs sub-pixel convolution layers to support various scale factors. A context-preserving 1D reorganization scheme is proposed to reduce the number of multipliers. Additional optimization technique is proposed to reduce the internal SRAM to store the input feature map. Thanks to the proposed optimizations along with additional techniques popular in hardware design, the number of multipliers and the size of SRAMs are reduced to 1/118.27 and 1/47.59, respectively, in comparison with a straightforward implementation of VDSR. The proposed CNN hardware synthesized with TSMC 65-nm CMOS technology, and it requires 8153.1K gate counts and 333.1kB SRAM to process a full-HD image. The proposed context-preserving 1D reorganization and memory reduction optimizations are also applicable to denoising and image classification without a significant performance degradation.

Optimized Hardware Design using Sobel and Median Filters for Lane Detection

In this paper, the image is received from the camera and the lane is sensed. There are various ways to detect lanes. Generally, the method of detecting edges uses a lot of the Sobel edge detection and the Canny edge detection. The minimum use of multiplication and division is used when designing for the hardware configuration. The images are tested using a black box image mounted on the vehicle. Because the top of the image of the used the black box is mostly background, the calculation process is excluded. Also, to speed up, YCbCr is calculated from the image and only the data for the desired color, white and yellow lane, is obtained to detect the lane. The median filter is used to remove noise from images. Intermediate filters excel at noise rejection, but they generally take a long time to compare all values. In this paper, by using addition, the time can be shortened by obtaining and using the result value of the median filter. In case of the Sobel edge detection, the speed is faster and noise sensitive compared to the Canny edge detection. These shortcomings are constructed using complementary algorithms. It also organizes and processes data into parallel processing pipelines. To reduce the size of memory, the system does not use memory to store all data at each step, but stores it using four line buffers. Three line buffers perform mask operations, and one line buffer stores new data at the same time as the operation. Through this work, memory can use six times faster the processing speed and about 33% greater quantity than other methods presented in this paper. The target operating frequency is designed so that the system operates at 50MHz. It is possible to use 2157fps for the images of 640by360 size based on the target operating frequency, 540fps for the HD images and 240fps for the Full HD images, which can be used for most images with 30fps as well as 60fps for the images with 60fps. The maximum operating frequency can be used for larger amounts of the frame processing.

A High-Throughput VLSI Architecture for Real-Time Full-HD Gradient Guided Image Filter

Guided image filtering has been applied widely in recent years as a solution to the ever-increasing demand of high-performance filtering, especially for real-time image/video processing. The lately proposed gradient domain guided image filter (GDGIF) is one of the typical works focusing on improving the quality of the filtering result of the original guided image filter (GIF), dealing with the halo-artifacts problem for edge-preserving smoothing. However, due to the involvement of global pixel values in the computation, high computation complexity, and additional complex operation, there is no existing VLSI design for the gradient guided filter. This paper presents a high-throughput VLSI architecture for real-time full-HD ( $1920\times 1080$ ) GDGIF. Four techniques are used in the newly proposed design to reduce the computation complexity and increase the processing throughput. First, a pre-processing stage is introduced to address the problem of global parameters required in the GDGIF. Second, multi-scale down-samplings are adopted for reducing circuit size and processing time. Then, a parallel structure operated at a higher frequency is used at the output stage to restore the full image size and to achieve high throughput at the same time. At last, approximated computation is introduced to eliminate speed bottleneck by simplifying the complex exponentiation operation. Based on the STM 65-nm CMOS technology, the implementation results show that the proposed architecture is able to support full-HD image filtering at a throughput above 75 frame/s, with a design area of 586314 $\mu \text{m}^{2}$ and a power consumption of 18.5 mW.

A fast image dehazing method that does not introduce color artifacts

We propose a method for color dehazing with four main characteristics: it does not introduce color artifacts, it does not depend on inverting any physical equation, it is based on models of visual perception, and it is fast, potentially real time. Our method converts the original input image to the HSV color space and works in the saturation and value domains by: (1) reducing the value component via a global constrained histogram flattening; (2) modifying the saturation component in consistency with the previous reduced value; and (3) performing a local contrast enhancement in the value component. Results show that our method competes with the state-of-the-art when dealing with standard hazy images, and outperforms it when dealing with challenging haze cases. Furthermore, our method is able to dehaze a FullHD image on a GPU in 90 ms.

LARGE SCALE TEXTURED MESH RECONSTRUCTION FROM MOBILE MAPPING IMAGES AND LIDAR SCANS

Abstract. The representation of 3D geometric and photometric information of the real world is one of the most challenging and extensively studied research topics in the photogrammetry and robotics communities. In this paper, we present a fully automatic framework for 3D high quality large scale urban texture mapping using oriented images and LiDAR scans acquired by a terrestrial Mobile Mapping System (MMS). First, the acquired points and images are sliced into temporal chunks ensuring a reasonable size and time consistency between geometry (points) and photometry (images). Then, a simple, fast and scalable 3D surface reconstruction relying on the sensor space topology is performed on each chunk after an isotropic sampling of the point cloud obtained from the raw LiDAR scans. Finally, the algorithm proposed in (Waechter et al., 2014) is adapted to texture the reconstructed surface with the images acquired simultaneously, ensuring a high quality texture with no seams and global color adjustment. We evaluate our full pipeline on a dataset of 17 km of acquisition in Rouen, France resulting in nearly 2 billion points and 40000 full HD images. We are able to reconstruct and texture the whole acquisition in less than 30 computing hours, the entire process being highly parallel as each chunk can be processed independently in a separate thread or computer.

Laparoscopic Typical and Atypical Locations of Sentinel Node Mapping with Indocyanine Green: Comparison of 2 Near-Infrared Fluorescence Systems

Study ObjectiveTo present our minimally invasive laparoscopic approach for sentinel lymph node (SLN) mapping with indocyanine green (ICG) using 2 fluorescence systems. DesignA step-by-step video description of the technique showing the most frequent typical and atypical location of SLNs (educational video). SettingLymph node staging in apparent confined endometrial cancer. PatientsWomen underwent SLN mapping in a minimally invasive setting. InterventionsLaparoscopic SLN mapping before comprehensive staging including simple hysterectomy, bilateral salpingo-oophorectomy, and pelvic and aortic bilateral lymphadenectomy in case of unilateral or no identification of SLNs. The PINPOINT 0 degree HD S1 SPY camera (PINPOINT Endoscopic Fluorescence Imaging System; NOVADAQ, Mississauga, ON, Canada) or the Full HD Image 1S with ICG camera (Karl Storz Endoscopy, Tuttlingen, Germany) were used for SLN detection [1,2]. The ICG powder was diluted to a final solution of 1.25 mg/mL of fluorescent dye. After the induction of general anesthesia, a total of 4 mL of the ICG solution was injected into the cervix at the 3 and 9 o'clock positions. Attention to the technical details is crucial to correctly identify SLNs that sometimes are located in atypical locations [3]. ConclusionBoth fluorescence systems are valid and applicable for SLN mapping in the case of apparent confined endometrial cancer. In our experience, the PINPOINT system seems to allow surgeons easier and faster nodal staging of the SLNs, particularly with the color-segmented fluorescence function activated, which can better discriminate between the lymphatic channels and the real SLNs [4,5].

Robust feature extraction algorithm suitable for real-time embedded applications

Smart cameras integrate processing close to the image sensor, so they can deliver high-level information to a host computer or high-level decision process. One of the most common processing is the visual features extraction since many vision-based use-cases are based on such algorithm. Unfortunately, in most of cases, features detection algorithms are not robust or do not reach real-time processing. Based on these limitations, a feature detection algorithm that is robust enough to deliver robust features under any type of indoor/outdoor scenarios is proposed. This was achieved by applying a non-textured corner filter combined to a subpixel refinement. Furthermore, an FPGA architecture is proposed. This architecture allows compact system design, real-time processing for Full HD images (it can process up to 44 frames/91.238.400 pixels per second for Full HD images), and high efficiency for smart camera implementations (similar hardware resources than previous formulations without subpixel refinement and without non-textured corner filter). For accuracy/robustness, experimental results for several real-world scenes are encouraging and show the feasibility of our algorithmic approach.

An OpenCL framework for high performance extraction of image features

Image features are widely used for object identification in many situations, including interpretation of data containing natural scenes captured by unmanned aerial vehicles. This paper presents a parallel framework to extract additive features (such as color features and histogram of oriented gradients) using the processing power of GPUs and multicore CPUs to accelerate the algorithms with the OpenCL language. The resulting features are available in device memory and then can be fed into classifiers such as SVM, logistic regression and boosting methods for object recognition. It is possible to extract multiple features with better performance. The GPU accelerated image integral algorithm speeds up computations up to 35x when compared to the single-thread CPU implementation in a test bed hardware. The proposed framework allows real-time extraction of a very large number of image features from full-HD images (better than 30 fps) and makes them available for access in coalesced order by GPU classification algorithms.

Design of a Binocular Pupil and Gaze Point Detection System Utilizing High Definition Images

This study proposes a novel binocular pupil and gaze detection system utilizing a remote full high definition (full HD) camera and employing LabVIEW. LabVIEW is inherently parallel and has fewer time-consuming algorithms. Many eye tracker applications are monocular and use low resolution cameras due to real-time image processing difficulties. We utilized the computer’s direct access memory channel for rapid data transmission and processed full HD images with LabVIEW. Full HD images make easier determinations of center coordinates/sizes of pupil and corneal reflection. We modified the camera so that the camera sensor passed only infrared (IR) images. Glints were taken as reference points for region of interest (ROI) area selection of the eye region in the face image. A morphologic filter was applied for erosion of noise, and a weighted average technique was used for center detection. To test system accuracy with 11 participants, we produced a visual stimulus set up to analyze each eye’s movement. Nonlinear mapping function was utilized for gaze estimation. Pupil size, pupil position, glint position and gaze point coordinates were obtained with free natural head movements in our system. This system also works at 2046 × 1086 resolution at 40 frames per second. It is assumed that 280 frames per second for 640 × 480 pixel images is the case. Experimental results show that the average gaze detection error for 11 participants was 0.76° for the left eye, 0.89° for right eye and 0.83° for the mean of two eyes.

The Smallest projection optics for the Vertical shaped Ultra short throw Projector

We have developed a smallest optics for a Vertical shaped Ultra short throw Projector which can project a Full HD (1920 × 1080 pixels) image. In order to realize the smallest optics, we have designed three specific optical parts. First, we have designed a negative distortion lens to minimize each optical parts. Second, we have put a flat mirror to bend optical paths. Finally, we have designed a freeform mirror to correct the distortion of a projected image. By using these parts, we have realized the smallest optics and excellent image quality for the Vertical shaped Ultra short throw Projector.