Pixel Coordinates Research Articles

Real-Time Information Fusion System Implementation Based on ARM-Based FPGA

In this study, an information fusion system displayed fusion information on a transparent display by considering the relationships among the display, background exhibit, and user’s gaze direction. We used an ARM-based field-programmable gate array (FPGA) to perform virtual–real fusion of this system as well as evaluated the virtual–real fusion execution speed. The ARM-based FPGA used Intel® RealsenseTM D435i depth cameras to capture depth and color images of an observer and exhibit. The image data was received by the ARM side and fed to the FPGA side for real-time object detection. The FPGA accelerated the computation of the convolution neural networks to recognize observers and exhibits. In addition, a module performed by the FPGA was developed for rapid registration between the color and depth images. The module calculated the size and position of the information displayed on a transparent display according to the pixel coordinates and depth values of the human eye and exhibit. A personal computer with GPU RTX2060 performed information fusion in ~47 ms, whereas the ARM-based FPGA accomplished it in 25 ms. Thus, the fusion speed of the ARM-based FPGA was 1.8 times faster than on the computer.

Computer vision for hunting stability inspection of high-speed trains

The existing online inspection methods for hunting motion of high-speed trains are mainly based on processing the acceleration of bogie frames, which cannot directly and reliably reflect the safety conditions. This paper proposes a novel computer vision (CV) method, named the virtual point tracking (VPT) method, to detect the hunting motion by processing the dynamic wheel-rail contact video. First, some virtual points are defined on each frame of the captured video, then the coordinates of these virtual points are automatically located by a deep learning model, and finally, the relative displacement between these coordinates is calculated. Two experiments demonstrate that the VPT method can accurately determine the hunting frequency and detect changes in the pixel coordinates corresponding to key positions of the wheel and the rail. Besides, a VPT method with Canny edge detection is introduced for comparison, and the results demonstrate that the VPT method is superior.

Protection of COVID-19 images using multiple elliptic curve cryptography

ABSTRACT COVID-19 pandemic has made the medical industry around the world under unprecedented and growing pressure. To preserve the integrity and protection of medical data for transmission and for secure diagnosis, this work suggests an encryption method for the protection of privacy of COVID-19 patient’s data by securing medical images based on elliptic curve cryptography. This is done through amalgamating three different types of stages by using elliptic curve cryptography. First, the coordinates of pixels of medical images are scrambled using elliptic curve and encrypt the coordinates of pixels, which help in decreasing the correlation between the image’s pixels. The second stage empowers the first stage by encrypting the group of pixels from previous outputs irrespective whether the pixel value lies on the curve or not based on the multiplication modulo operation of elliptic curve cryptography. Finally, the group of pixels from the previous output is encrypted by XORing with the key generated from ECC DHKE (Elliptic Curve Cryptography Diffie–Hellman Key Exchange). The NIST test suite examines and verifies the statistical characteristics of the binary sequences for 2 10 bits and 2 16 bits that are created by cryptographic random or pseudorandom number generators. The outcomes of the experiments indicate how the suggested approach satisfies the security requirements of COVID-19 images of patients in terms of high effectiveness in encryption quality, resilient against many statistical analyses and attacks and also performs better than some recent schemes. This work also evaluated comparisons of quantitative, differential, time consumptions and computational complexity with several recent schemes.

NDMFCS: An automatic fruit counting system in modern apple orchard using abatement of abnormal fruit detection

Automatic fruit counting is an important task for growers to estimate yield and manage orchards. Although many deep-learning-based fruit detection algorithms have been developed to improve performance of automatic fruit counting systems, abnormal fruit detection has often been caused by these algorithms detecting non-target fruits that have similar growth characteristics to target fruits. For abnormal fruit detection, detected fruits in the back row of the tree were defined as DFBRT, while detected fruits on the ground were defined as DFG. Both of them would result in a higher number of fruits counting than the ground truth. This study proposes an automatic fruit counting system called NDMFCS (Normal Detection Matched Fruit Counting System) to solve this problem for improving fruit counting accuracy in modern apple orchard. NDMFCS consists of three sub-systems, i.e. object detection based on You Only Look Once Version 4-tiny (YOLOv4-tiny), abatement of abnormal fruit detection based on threshold, and fruit counting based on trunk tracking and identity document (ID) assignment. YOLOv4-tiny was selected to implement detection of fruits and trunks, whose output is confidence and pixel coordinates of detected object. The DFBRT and DFG were abated by thresholds to improve detection performance of fruit. This meant that detected fruits were removed when their distance from camera is further than a distance threshold or the confidence of fruit detection is less than a confidence threshold. Finally, fruit counting was implemented by trunk tracking and ID assignment, where each fruit was assigned a unique tracking ID. Results on 10 sets of original videos indicated that average fruit detection precision was improved from 89.1% to 93.3% after abatement of abnormal fruit detection. Also, Multiple Object Tracking Accuracy and Multiple Object Tracking Precision were improved on average by 4.2% and 3.3%, respectively, while average ID Switch Rate was decreased on average by 1.1%. And average fruit counting accuracy was improved to 95.0% by 4.2%. Coefficient of determination (R2) was 0.97, which indicated the number of fruits counted by NDMFCS was near to the ground truth. These results demonstrate that the abatement of abnormal fruit detection can improve performance of apple counting, which has the potential to provide a technical support for estimating fruit yield in modern apple orchards.

RGB-D 데이터를 이용한 비전 기반 로봇 간 상대 위치 및 자세 추정 기법 연구

In this paper, a vision-based robot-to-robot relative pose estimation method is proposed for markerless wheeled mobile robots using an RGB-D camera. The proposed method comprises three parts: (i) object detection using a machine vision technique with RGB data, (ii) relative position estimation averaging over the point cloud with outlier rejection, and (iii) relative orientation estimation using point cloud scan matching through iterative closest point (ICP) methods. The neighboring or target robots are detected using YOLOv3-tiny so that the position of the robot can be determined in pixel coordinates. To estimate the 3D position of the target robot in the camera coordinates of the ego-robot, the object detection results are used along with the depth information and the precalibrated pinhole camera model. To estimate the orientation of the target robot, a point cloud is generated using the depth image, downsampled, and then used for scan matching by using ICP methods. For experimental verification, the proposed relative pose estimation method was implemented as a robot operating system package and tested on a small-scale differential-driving wheeled mobile robot, TurtleBot3; the estimation results were compared with the ground truth in real-world environments.

Nonlinear optimization considering target feature points for bitelecentric camera calibration: experimental study.

The telecentric camera has found extensive application in microscopy imaging due to its remarkable attributes of maintaining constant magnification and minimal distortion within its depth of field. In telecentric imaging technology, the accuracy of measurements frequently hinges upon the calibration precision of the telecentric camera. In real-world scenarios, the shallow depth of field characteristic of telecentric cameras often leads to out-of-focus targets during the capturing process, which in turn results in the inability to accurately extract pixel coordinates of feature points, making it difficult for optimization algorithms to converge to the optimal value. We propose a nonlinear optimization algorithm based on pixel coordinates of optimized feature points for bitelecentric cameras. Incorporating pixel coordinates into the optimization process yields the theoretically optimal solution based on bitelecentric camera model. The obtained pixel coordinates are used for second initial value estimation, followed by the optimization of all parameters. Compared to existing methods, the proposed approach significantly reduces reprojection errors under both blurry and clear target conditions. Experimental results demonstrate superior performance in processing blurry defocused images.

Robot teaching system based on hand-robot contact state detection and motion intention recognition

This paper presents a robot teaching system based on hand-robot contact state detection and human motion intent recognition. The system can detect the contact state of the hand-robot joint and extracts motion intention information from the human surface electromyography (sEMG) signals to control the robot's motion. First, a hand-robot contact state detection method is proposed based on the fusion of the virtual robot environment with the physical environment. With the use of a target detection algorithm, the position of the human hand in the color image of the physical environment can be identified and its pixel coordinates can be calculated. Meanwhile, the synthetic images of the virtual robot environment are combined with those of the physical robot scene to determine whether the human hand is in contact with the robot. Besides, a human motion intention recognition model based on deep learning is designed to recognize human motion intention with the input of sEMG signals. Moreover, a robot motion mode selection module is built to control the robot for single-axis motion, linear motion, or repositioning motion by combining the hand-robot contact state and human motion intention. The experimental results indicate that the proposed system can perform online robot teaching for the three motion modes.

Velocity Estimation for Space Infrared Dim Targets Based on Multi-Satellite Observation and Robust Locally Weighted Regression

Velocity estimation of space moving targets is a key part of space situational awareness. However, most of the existing methods do not consider the satellite observation process, and the performance mainly depends on the preset target motion state, which has great limitations. To accurately obtain the motion characteristics of space infrared dim targets in space-based infrared detection, a velocity estimation method based on multi-satellite observation and robust locally weighted regression is proposed. Firstly, according to parameters such as satellite position, satellite attitude angle, and sensor line of sight, the overall target observation model from the sensor coordinate frame to the Earth-centered inertial coordinate frame is established, and the pixel coordinates of the target imaging point are extracted using the gray-weighted centroid method. Then, combined with the least squares criterion, the position sequence of the space target is obtained. Finally, a robust locally weighted regression operation is performed on the target position sequence to estimate the velocity. This study verified the feasibility of the proposed method through simulation examples, with the results showing that the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) of the method were only 0.0733 m/s and 1.6640 m/s without measurement error. Moreover, the velocity estimation accuracy was better than that of other methods in most scenarios. In addition, the estimation accuracy under the impact of various measurement errors was analyzed, and it was found that the pixel coordinate extraction error had the greatest impact on velocity estimation accuracy. The proposed method provides a technical basis for the recognition of space infrared dim moving targets.

Calibration method for fringe projection profilometry based on rational function lens distortion model

The calibration of the fringe projection profilometry (FPP) system is susceptible to the camera lens distortion. This paper presents a calibration method based on the rational function lens distortion model. In this method, we integrate the rational function into the mathematical expression of the out-of-plane height to mitigate the impact of lens distortion. With arbitrarily arranged measurement components, we can flexibly measure the three-dimensional (3D) shape of objects with no expensive auxiliaries, high-precision setups, or complicated projector calibration. First, we project modulated sinusoidal fringe patterns with high frequency to the concentric circle grids calibration board to accurately estimate the unwrapped phase value, the pixel coordinate, and the out-of-plane height at each feature point. Then, the 3D coordinates can be recovered by determining the coefficients of the high order phase, coordinate, and height (PCH) mapping. Experiments on computer simulation and real data demonstrate that the modified calibration contributes to the flexibility and accuracy of the 3D shape measurement.

LEO Satellite Navigation Based on Optical Measurements of a Cooperative Constellation

Autonomous, anti-jamming, and high-precision satellite navigation are of great importance to current and future space technologies. This paper proposes a cooperative constellation navigation system for low Earth orbit (LEO) satellites that use only the optical measurements of cooperative satellites. Based on photometry, an optical transmission link model of the system is built. With the pixel coordinates of the cooperative satellites on the optical images, the line of sight (LoS) vectors of the cooperative satellites with respect to the LEO spacecraft are first calculated, and a single-point positioning method based on the LoS vectors’ inner products is proposed. The single-point positioning results are then fed into a least square batch filter to estimate a high-precision spacecraft orbit. Simulations are conducted to evaluate the potential navigation accuracy. With a cooperative satellite ephemeris error of 100 m and an optical measurement noise level of 5 arcsecs, position accuracies of single-point positioning and dynamic orbit determination in the order of hundreds of meters and eight meters, respectively, are realized. In addition, the influences of the orbital altitude of the cooperative constellation, the ephemeris error of the cooperative satellite, the noise level of the optical measurements, and the Earth’s gravitational model on navigation accuracy are investigated via comparative simulations.

An attitude estimation algorithm for the telescopic arm of the boarding bridge based on YOLOv5 and EPnP

Aiming at the attitude estimation of the telescopic arm of the boarding bridge in the process of docking with the offshore wind turbine, an attitude estimation algorithm for the telescopic arm of the boarding bridge based on YOLOv5 and EPnP is proposed in this paper. YOLOv5 algorithm is used to detect four marks in the offshore wind turbine logo image, and then the EPnP algorithm is used to solve attitude angles of the telescopic arm relative to the landing port of the offshore wind turbine according to the 2D pixel coordinates of the center point of each mark. The experimental results show that the attitude estimation error of the proposed algorithm is less than 0.4° and the data update frequency of the algorithm implemented in NVIDIA Jetson AGX Xavier is 25Hz, which meets the real-time and accuracy requirements for the attitude detection of the boarding bridge telescopic arm.

Pixel Coordinates for Obtaining 3D Body Measurements from 2D Images

In this research, a new method developed with pixel coordinates is presented to obtain 3D body measurements from 2D images, to eliminate the need of the body to take measurements and to solve unfit problem in e-shopping. Front and side photographs are processed with new developed programmes named PixelList1.0 and Python 2.0. The sharp silhouettes, presented in authors’ another publication, are benefited with algorithms to reach pixel coordinates of feature points. Number of pixels in between feature points are saved, absolute values of differences consecutively are calculated, and 3D body measurements in pixels are obtained with different mathematical models. A conversion coefficient is introduced and size of a pixel is calculated accurately, 0.239 cm/pixel. 3D body measurements in centimeters are obtained by multiplying number of pixels by the size of one pixel in circle, ellipse, and hippodrome circumference equations. An overall accuracy rate of 93 % is concluded with the ellipse circumference equation.

Wind Speed Measurement via Visual Recognition of Wind-Induced Waving Light Stick Target

Wind measurement in confined spaces is a challenge due to the influence of the dimensions of anemometers in intrusive flow-field measurements where the anemometer probes directly contact and influence the near-probe flow field. In this work, a new wind speed detection methodology is proposed based on wind-induced motion of a stick via vision-based recognition. The target’s displacement in pixel coordinates is mapped to its angular displacement in world coordinates to derive wind speed and direction information by applying the calibration coefficients. Simulation experiments were carried out to validate the model, the error of which was within an angular displacement of 4.0° and 3.0° for wind speed and direction detections, respectively. When applied to the measurement of wind speed in the inner equipment cabin of a stationary high-speed train, the error was within ±1.1 m/s in terms of average RMSE. Thus, the proposed method provides an accurate and economic option for monitoring 2D wind in a confined space.

Cross-Modal Image Registration via Rasterized Parameter Prediction for Object Tracking

Object tracking requires heterogeneous images that are well registered in advance, with cross-modal image registration used to transform images of the same scene generated by different sensors into the same coordinate system. Infrared and visible light sensors are the most widely used in environmental perception; however, misaligned pixel coordinates in cross-modal images remain a challenge in practical applications of the object tracking task. Traditional feature-based approaches can only be applied in single-mode scenarios, and cannot be well extended to cross-modal scenarios. Recent deep learning technology employs neural networks with large parameter scales for prediction of feature points for image registration. However, supervised learning methods require numerous manually aligned images for model training, leading to the scalability and adaptivity problems. The Unsupervised Deep Homography Network (UDHN) applies Mean Absolute Error (MAE) metrics for cost function computation without labelled images; however, it is currently inapplicable for cross-modal image registration. In this paper, we propose aligning infrared and visible images using a rasterized parameter prediction algorithm with similarity measurement evaluation. Specifically, we use Cost Volume (CV) to predict registration parameters from coarse-grained to fine-grained layers with a raster constraint for multimodal feature fusion. In addition, motivated by the utilization of mutual information in contrastive learning, we apply a cross-modal similarity measurement algorithm for semi-supervised image registration. Our proposed method achieves state-of-the-art performance on the MS-COCO and FLIR datasets.

A new SINS/RCNS integrated navigation method based on star pixel coordinates

A new SINS/RCNS integrated navigation method based on star pixel coordinates

Spatial Recognition: A Temporal Tale of Multiple Window Spatial Registration.

The performance ability of position logic circuitry of gamma-camera to superimpose the X-Y gains obtained from different pulse height analyzers is multiple window spatial registration (MWSR). This study was conducted to evaluate the feasibility of dual-tracer method using 99mTc and 131I to assess MWSR. The procedure was carried out on a dual-head gamma-camera with rectangular NaI (Tl) crystal using a point source containing 99mTc pertechnetate and 131I-NaI. Pixel coordinates, full width at half-maximum of point spread function at 364 keV and 140 keV, and pixels with maximum count of images for dual detectors were analyzed and compared. The experiment demonstrated an excellent correlation between the X and Y coordinates identified on 364 keV and 140 keV images (Spearman's coefficient of rank correlation: 0.9, P < 0.0001). Intraclass correlation of studies performed on two separate occasions showed a good correlation (k = 0.87) demonstrating the reproducibility of the procedure. Dual-tracer method for evaluation of MWSR is a reproducible, accurate, and logical alternative to the routine National Electrical Manufacturers Association procedure.

Pixel-Coordinate-Induced Human Pose High-Precision Estimation Method

Accurately estimating human pose is crucial for providing feedback during exercises or musical performances, but the complex and flexible nature of human joints makes it challenging. Additionally, traditional methods often neglect pixel coordinates, which are naturally present in high-resolution images of the human body. To address this issue, we propose a novel human pose estimation method that directly incorporates pixel coordinates. Our method adds a coordinate channel to the convolution process and embeds pixel coordinates into the feature map, while also using coordinate attention to capture position- and structure-sensitive features. We further reduce the network parameters and computational cost by using small-scale convolution kernels and a smooth activation function in residual blocks. We evaluate our model on the MPII Human Pose and COCO Keypoint Detection datasets and demonstrate improved accuracy, highlighting the importance of directly incorporating coordinate location information in position-sensitive tasks.

Optical plasma boundary detection and its reconstruction on EAST tokamak

Plasma boundary detection and reconstruction are important not only for plasma operation but also for plasma facing materials. Traditional methods, for example, EFIT code, which is constrained by electromagnetic measurement, and is very challenging for detecting the plasma boundary in long-pulse burning plasma devices such as ITER. A novel algorithm for the reconstruction of the plasma boundary using one visible camera has been developed on experimental advanced superconducting tokamak (EAST) for fusion reactors. A U-Net convolutional neural network was used to identify the plasma boundary and the pixel coordinates of the boundary points were fitted with EFIT via the XGBoost model. This algorithm can transform the boundary from the image plane to the poloidal plane of the Tokamak based on machine learning without traditional spatial calibration, and then the reconstruction of the plasma configuration shall be realized based on a monocular visible light camera. The reconstruction accuracy of this algorithm is relatively high. The average error on the test set was only 7.36 mm (<1 cm) and satisfied the accuracy requirements of control for EAST tokamak. This result can contribute to the development of the plasma boundary reconstruction and operation based on one visible camera.

Umbrella Trees for Nesting Problem

The problem of irregular-shaped pieces representation in order to minimize the nesting time is considered. A representation in the form of umbrella tree with a square crown consisting of tricolor nodes and bicolor leaves equidistant from the root is proposed. The construction of an umbrella tree from a image based on the transformation of Cartesian pixel coordinates into reference coordinates of tree leaves is shown. The translation and rotation of pieces functions on umbrella trees with quadratic estimates of computational complexity are determined. Examples of rounding coordinates heuristics and the experimental values of the threshold recoloring leaves during rotating a piece are given. The quadratic problem of intersection localization of the slab contents and the piece is reduced to the evaluating of color mutual exclusivity predicate of the pairs of nodes of the slab tree and the piece tree with linearrhythmic estimation of time in the applied cases. A pair of nodes is mutually exclusive if at least one of the nodes is white, and it isn't mutually exclusive if both are black. In other cases a pair of nodes is considered mutually exclusive if all four pairs of child nodes are mutually exclusive. The price of reducing the computational complexity of intersection localization is an increase of the required memory size by 33 % compared to the raster representation. Umbrella trees make it possible to use photo and video frames of pieces and slabs without converting them into a vector representation.

Relative visual navigation around an unknown and uncooperative spacecraft

A Robotic Active Debris Removal (ADR) mission requires close-proximity operations for capturing potentially tumbling, uncooperative targets and disposing of them. These operations rely on accurate relative navigation between the chaser and the target to safely perform the mission. In the case of a known piece of debris that has not been designed to be serviced or disposed of, the knowledge of its shape factor allows relative pose determination. In the case of unknown space debris, however, no relative pose determination is possible as no reference is available beforehand. In this paper, we derive an onboard software architecture for estimating the shape and dynamical model of an unknown, uncooperative space debris, and perform tracking, using features detected in the image and the associated depth. Firstly, an Error-State Kalman filter (ESKF) is used to estimate the chaser trajectory and attitude as well as the target shape in an arbitrarily fixed target frame. The next step in the architecture is to apply a Gaussian Processes (GPs) algorithm to model the rotational dynamics of the debris, allowing an accurate attitude propagation independently of its tumbling mode. Finally, a second ESKF is instantiated using existing knowledge to track the position and attitude of the target. This work assumes the pre-existence of a feature detection, tracking and matching algorithm, and fusing data from a depth sensor allowing the retrieval of pixel coordinates and depth for each detected feature. The proposed architecture has been tested and validated in a computer-simulated environment with parameters representative of a real-world ADR mission. Results from over 100 simulations show a mean position and attitude error of 0.3 m and 1.5°, respectively.