Fiducial Marker System Research Articles

Design and development of robotic collaborative system for automated construction of reciprocal frame structures

AbstractRobotic technologies have shown their potential to improve efficiency, precision, and safety for construction tasks. In this paper, the concept of design for robotic construction (DfRC) is introduced, and robotic collaborative systems are developed for the construction of load‐carrying structures. An automated structural assembly was achieved and demonstrated through robotics with a preference for reciprocal frame (RF) structures. Key innovations include the use of magnetic temporary connections to minimize offsets of mobile robots and linear actuators as temporary supports during construction. Furthermore, specific procedures are formulated to navigate the robots and to target and install the components using a fiducial marker system and simultaneous localization and mapping packages, with consideration of structural deformation during construction due to self‐weight. The successful assembly of a 4.5‐m span RF structure demonstrates the potential of DfRC and the proposed robotic collaborative system in the automated construction of load‐carrying structures.

Automated Computer Vision-Enabled Manufacturing of Nanowire Devices.

We present a high-throughput method for identifying and characterizing individual nanowires and for automatically designing electrode patterns with high alignment accuracy. Central to our method is an optimized machine-readable, lithographically processable, and multi-scale fiducial marker system─dubbed LithoTag─which provides nanostructure position determination at the nanometer scale. A grid of uniquely defined LithoTag markers patterned across a substrate enables image alignment and mapping in 100% of a set of >9000 scanning electron microscopy (SEM) images (>7 gigapixels). Combining this automated SEM imaging with a computer vision algorithm yields location and property data for individual nanowires. Starting with a random arrangement of individual InAs nanowires with diameters of 30 ± 5 nm on a single chip, we automatically design and fabricate >200 single-nanowire devices. For >75% of devices, the positioning accuracy of the fabricated electrodes is within 2 pixels of the original microscopy image resolution. The presented LithoTag method enables automation of nanodevice processing and is agnostic to microscopy modality and nanostructure type. Such high-throughput experimental methodology coupled with data-extensive science can help overcome the characterization bottleneck and improve the yield of nanodevice fabrication, driving the development and applications of nanostructured materials.

Intensity Image-Based LiDAR Fiducial Marker System

The fiducial marker system for LiDAR is crucial for the robotic application but it is still rare to date. In this paper, an Intensity Image-based LiDAR Fiducial Marker (IILFM) system is developed. This system only requires an unstructured point cloud with intensity as the input and it has no restriction on marker placement and shape. A marker detection method that locates the predefined 3D fiducials in the point cloud through the intensity image is introduced. Then, an approach that utilizes the detected 3D fiducials to estimate the LiDAR 6-DOF pose that describes the transmission from the world coordinate system to the LiDAR coordinate system is developed. Moreover, all these processes run in real-time (approx 40 Hz on Livox Mid-40 and approx 143 Hz on VLP-16). Qualitative and quantitative experiments are conducted to demonstrate that the proposed system has similar convenience and accuracy as the conventional visual fiducial marker system. The codes and results are available at: https://github.com/York-SDCNLab/IILFM.

MechaTag: A Mechanical Fiducial Marker and the Detection Algorithm

Fiducial markers are fundamental components of many computer vision systems that help, through their unique features (e.g., shape, color), a fast localization of spatial objects in unstructured scenarios. They find applications in many scientific and industrial fields, such as augmented reality, human-robot interaction, and robot navigation. In order to overcome the limitations of traditional paper-printed fiducial markers (i.e. deformability of the paper surface, incompatibility with industrial and harsh environments, complexity of the shape to reproduce directly on the piece), we aim at exploiting existing, or additionally fabricated, structural features on rigid bodies (e.g., holes), developing a fiducial mechanical marker system called MechaTag. Our system, endowed with a dedicated algorithm, is able to minimize recognition errors and to improve repeatability also in case of ill boundary conditions (e.g., partial illumination). We assess MechaTag in a pilot study, achieving a robustness of fiducial marker recognition above 95% in different environment conditions and position configurations. The pilot study was conducted by guiding a robotic platform in different poses in order to experiment with a wide range of working conditions. Our results make MechaTag a reliable fiducial marker system for a wide range of robotic applications in harsh industrial environments without losing accuracy of recognition due to the shape and material.

Five Day Accelerated Partial Breast Irradiation (APBI) Using Stereotactic Body Radiation Therapy (SBRT) in Stage 0-II Breast Cancer: A Report of 218 Cases With Up to 39 Month Follow-Up

Randomized trials in selected early-stage breast cancer patients with up to 10-year follow-up have proven that Accelerated Partial Breast Irradiation (APBI) given via High Dose Rate (HDR) implant bid in 5 days is equivalent to whole breast irradiation (WBI) given q.d. in 5-6 weeks in regard to breast tumor local recurrence (LR). However, complications with APBI implant in a Medicare database review have been significant, with 3.95% of women requiring Mastectomy, 16.2% developing infections, and another 16.3% experiencing non-infection complications including rib fractures, fat necrosis, and breast pain. Recently APBI using non-invasive Intensity Modulated Radiation Therapy (IMRT) or SBRT given q.d. in 5 fractions has been shown in another randomized trial with 10-year follow-up to be equivalent to q.d. WBI in 6 weeks, with respect to LR. IMRT/SBRT was superior in regard to acute effects, late effects, and cosmesis. In the randomized clinical trial of APBI IMRT/SBRT, the Clinical Target Volume (CTV) was defined by the injection of individual fiducial markers bordering the surgical cavity. At our institution, we have used a bioabsorbable fiducial marker system to localize the CTV for SBRT. We sought to confirm the APBI IMRT/SBRT results with this simpler less labor-intensive fiducial placement system.Between 2017 and 2021, 218 patients have undergone SBRT targeted to a bioabsorbable fiducial marker defined CTV with the walls of the surgical cavity sewn to the fiducial marker. Eligible patients were older than age 40, had tumor sizes ≤ 3 cm, negative surgical margins, and negative sentinel node dissections. SBRT dose was 30 Gy given in 5 fractions. Dose Constraints were as follows: V-30 Gy < 105%, Ipsilateral Breast V-15 Gy < 50%, Ipsilateral Lung V-10 Gy < 20%, Contralateral Lung V-5 Gy < 10%, Heart V-3 Gy < 20%, Contralateral Breast Dmax < 2 Gy and Skin Dmax < 27 Gy. The Planning Target Volume (PTV) ranged from 27 to 355 cc with a median of 80 cc. PTV = CTV + 1-2 cm.Follow-up ranged from 1-39 months with a median of 20 months. LR has been 0% (0/218). There have been no skin reactions, seromas, or infections. Four (2.1%) patients developed pain around the fiducial marker site. This resolved within 2 days on a short course of steroids in all cases. Cosmetic results as rated by the Surgeon, Radiation Oncologist, and Nurse, were rated excellent in 99.0% (216/218) of cases.Non-invasive APBI with SBRT given q.d. over 5 days targeted to bioabsorbable fiducial marker has resulted in LR, complications, and cosmetic results which compare favorably to invasive APBI given bid with HDR implant. At last follow-up, there have been no LR, skin reactions, or significant complications. Cosmesis has been excellent in 99.0% of patients.

DeepFormableTag

Fiducial markers have been broadly used to identify objects or embed messages that can be detected by a camera. Primarily, existing detection methods assume that markers are printed on ideally planar surfaces. The size of a message or identification code is limited by the spatial resolution of binary patterns in a marker. Markers often fail to be recognized due to various imaging artifacts of optical/perspective distortion and motion blur. To overcome these limitations, we propose a novel deformable fiducial marker system that consists of three main parts: First, a fiducial marker generator creates a set of free-form color patterns to encode significantly large-scale information in unique visual codes. Second, a differentiable image simulator creates a training dataset of photorealistic scene images with the deformed markers, being rendered during optimization in a differentiable manner. The rendered images include realistic shading with specular reflection, optical distortion, defocus and motion blur, color alteration, imaging noise, and shape deformation of markers. Lastly, a trained marker detector seeks the regions of interest and recognizes multiple marker patterns simultaneously via inverse deformation transformation. The deformable marker creator and detector networks are jointly optimized via the differentiable photorealistic renderer in an end-to-end manner, allowing us to robustly recognize a wide range of deformable markers with high accuracy. Our deformable marker system is capable of decoding 36-bit messages successfully at ~29 fps with severe shape deformation. Results validate that our system significantly outperforms the traditional and data-driven marker methods. Our learning-based marker system opens up new interesting applications of fiducial markers, including cost-effective motion capture of the human body, active 3D scanning using our fiducial markers' array as structured light patterns, and robust augmented reality rendering of virtual objects on dynamic surfaces.

Application of a Vision-Based Single Target on Robot Positioning System.

In this paper, we propose a Circular-ring visual location marker based on a global image-matching model to improve the positioning ability in the fiducial marker system of a single-target mobile robot. The unique coding information is designed according to the cross-ratio invariance of the projective theorem. To verify the accuracy of full 6D pose estimation using the Circular-ring marker, a 6 degree of freedom (DoF) robotic arm platform is used to design a visual location experiment. The experimental result shows in terms of small resolution images, different size markers, and long-distance tests that our proposed robot positioning method significantly outperforms AprilTag, ArUco, and Checkerboard. Furthermore, through a repeatable robot positioning experiment, the results indicated that the proposed Circular-ring marker is twice as accurate as the fiducial marker at 2–4 m. In terms of recognition speed, the Circular-ring marker processes a frame within 0.077 s. When the Circular-ring marker is used for robot positioning at 2–4 m, the maximum average translation error of the Circular-ring marker is 2.19, 3.04, and 9.44 mm. The maximum average rotation error is also 1.703°, 1.468°, and 0.782°.

Towards manufacturing robotics accuracy degradation assessment: A vision-based data-driven implementation

In this manuscript we report on a vision-based data-driven methodology for industrial robot health assessment. We provide an experimental evidence of the usefulness of our methodology on a system comprised of a 6-axis industrial robot, two monocular cameras and five binary squared fiducial markers. The fiducial marker system permits to accurately track the deviation of the end-effector along a fixed non-trivial trajectory. Moreover, we monitor the trajectory deflection using three gradually increasing weights attached to the end-effector. When the robot is loaded with the maximum allowed payload, a deviation of 0.77mm is identified in the Z-coordinate of the end-effector. Tracing trajectory information, we train five supervised learning regression models. Such models are afterwards used to predict the deviation of the end-effector, using the pose estimation provided by the visual tracking system. As a result of this study, we show that this procedure is a stable, robust, rigorous and reliable tool for robot trajectory deviation estimation and it even allows to identify the mechanical element producing non-kinematic errors.

Augmented Reality Approach for Marker-based Posture Measurement on Smartphones.

Marker tracking for postural and range of motion (ROM) measurements transcends multiple disciplines (e.g., healthcare, ergonomics, engineering). A viable real-time mobile application is currently lacking for measuring limb angles and body posture. To address this need, a novel Android smartphone augmented-reality-based application was developed using the AprilTag2 fiducial marker system. To evaluate the app, two markers were printed on paper and attached to a wall. A Samsung S6 mobile phone was fixed on a tripod, parallel to the wall. The smartphone app tracked and recorded marker orientation and 2D position data in the camera frame, from front and rear cameras, for different smartphone placements. The average error between mobile phone and measured angles was less than 1 degree for all test settings (back camera=0.29°, front camera=0.33°, yaw rotation=0.75°, tilt rotation=0.22°). The average error between mobile phone and measured distance was less than 4 mm for all test settings (back camera=1.8 mm, front camera=2.5 mm, yaw rotation=3 mm, tilt rotation=3.8 mm). Overall, the app obtained valid and reliable angle and distance measurements with smartphone positions and cameras that would be expected in practice. Thus, this app is viable for clinical ROM and posture assessments.

ACMarker: Acoustic Camera-Based Fiducial Marker System in Underwater Environment

ACMarker is an acoustic camera-based fiducial marker system designed for underwater environments. Optical camera-based fiducial marker systems have been widely used in computer vision and robotics applications such as augmented reality (AR), camera calibration, and robot navigation. However, in underwater environments, the performance of optical cameras is limited owing to water turbidity and illumination conditions. Acoustic cameras, which are forward-looking sonars, have been gradually applied in underwater situations. They can acquire high-resolution images even in turbid water with poor illumination. We propose methods to recognize a simply designed marker and to estimate the relative pose between the acoustic camera and the marker. The proposed system can be applied to various underwater tasks such as object tracking and localization of unmanned underwater vehicles. Simulation and real experiments were conducted to test the recognition of such markers and pose estimation based on the markers.

TopoTag: A Robust and Scalable Topological Fiducial Marker System.

Fiducial markers have been playing an important role in augmented reality (AR), robot navigation, and general applications where the relative pose between a camera and an object is required. Here we introduce TopoTag, a robust and scalable topological fiducial marker system, which supports reliable and accurate pose estimation from a single image. TopoTag uses topological and geometrical information in marker detection to achieve higher robustness. Topological information is extensively used for 2D marker detection, and further corresponding geometrical information for ID decoding. Robust 3D pose estimation is achieved by taking advantage of all TopoTag vertices. Without sacrificing bits for higher recall and precision like previous systems, TopoTag can use full bits for ID encoding. TopoTag supports tens of thousands unique IDs and easily extends to millions of unique tags resulting in massive scalability. We collected a large test dataset including in total 169,713 images for evaluation, involving in-plane and out-of-plane rotation, image blur, different distances, and various backgrounds, etc. Experiments on the dataset and real indoor and outdoor scene tests with a rolling shutter camera both show that TopoTag significantly outperforms previous fiducial marker systems in terms of various metrics, including detection accuracy, vertex jitter, pose jitter and accuracy, etc. In addition, TopoTag supports occlusion as long as the main tag topological structure is maintained and allows for flexible shape design where users can customize internal and external marker shapes. Code for our marker design/generation, marker detection, and dataset are available at http://herohuyongtao.github.io/research/publications/topo-tag/.

STag: A stable fiducial marker system

Fiducial markers provide better-defined features than the ones naturally available in the scene. For this reason, they are widely utilized in computer vision applications where reliable pose estimation is required. Factors such as imaging noise and subtle changes in illumination induce jitter on the estimated pose. Jitter impairs robustness in vision and robotics applications, and deteriorates the sense of presence and immersion in AR/VR applications. In this paper, we propose STag, a fiducial marker system that provides stable pose estimation. STag is designed to be robust against jitter factors, thus sustains pose stability better than the existing solutions. This is achieved by utilizing geometric features that can be localized more repeatably. The outer square border of the marker is used for detection and homography estimation. This is followed by a novel homography refinement step using the inner circular border. After refinement, the pose can be estimated stably and robustly across viewing conditions. These features are demonstrated with a comprehensive set of experiments, including comparisons with the state of the art fiducial marker systems.

Enhancing re-detection efficacy of defects on blank wafers using stealth fiducial markers

To qualify tools of semiconductor manufacturing, particles unintentionally deposited in these tools are characterized using blank wafers. With fast optical inspection tools one can quickly localize these particle defects. An example is TNO's Rapid Nano, which operates in optical dark field. The next step is defect review for further defect characterization. When the blank wafers are transferred to another tool, e.g. a SEM or an AFM the absolute defect position information is lost. Therefore, the re-detection of the defects in the review tool is time consuming. To enhance the re-detection speed, a fiducial marker system can be used that couples the coordinates of the fast inspection tool to the coordinates of the characterization (review) tool.In this work such a fiducial marker system was designed and validated. The influences of the height and the composition of the fiducial markers on the performance of the marker system were investigated using finite element analysis (by COMSOL) and experiments. The optimized fiducial markers are very visible in optical bright field and in SEM, while almost invisible (“stealth”) in optical dark field. These properties make the markers both easily visible and accurately localizable in the characterization tools. The stealth fiducial marker system was fabricated and validated by re-detecting programmed test defects on a blank wafer. The experimental results are compared to a Monte Carlo simulation that takes into account the uncertainties in the coordinate transformation and localization of the test defects.Our results show that a fiducial marker system greatly enhances the re-detection efficacy of defects on blank wafers. Using the fiducial marker system, 100% of the test defects were re-detected in SEM and AFM. A single 7×7μm2 SEM image suffices to meet the ITRS requirement for particles as small as 70nm in diameter.

Guidance, Navigation, and Control for Docking of Two Cubic Blimps

This paper focuses on the development of guidance, navigation and control for the docking of two cubic blimps, named Tryphons. The chaser and target Tryphons are assumed to be brought sufficiently close together through a rendezvous stage, where the docking stage then begins and the target is kept stationary. An ARtag fiducial marker system is used for position based visual servoing control with a glideslope guidance algorithm for soft docking. The physical docking of the Tryphons is achieved through contact of the protruder and receiver ends of the two electromagnetic docking mechanisms. Simulations of the docking process are carried out in the open source simulation software Gazebo, using Robot Operating System (ROS) and virtual sensors.

Automatic generation and detection of highly reliable fiducial markers under occlusion

This paper presents a fiducial marker system specially appropriated for camera pose estimation in applications such as augmented reality and robot localization. Three main contributions are presented. First, we propose an algorithm for generating configurable marker dictionaries (in size and number of bits) following a criterion to maximize the inter-marker distance and the number of bit transitions. In the process, we derive the maximum theoretical inter-marker distance that dictionaries of square binary markers can have. Second, a method for automatically detecting the markers and correcting possible errors is proposed. Third, a solution to the occlusion problem in augmented reality applications is shown. To that aim, multiple markers are combined with an occlusion mask calculated by color segmentation. The experiments conducted show that our proposal obtains dictionaries with higher inter-marker distances and lower false negative rates than state-of-the-art systems, and provides an effective solution to the occlusion problem.

Low-Cost, High-Performance Monocular Vision System for Air Bearing Table Attitude Determination

Attitude determination is one of the most important prerequisites for the implementation of an air bearing table. In this context, computer vision is shown to be an enabling technology; a cost-effective monocular camera vision system, including hardware setup and navigation software, has been developed to determine the attitude of the air bearing table. Both infrared light-emitting diodes and filter are selected to simplify image processing, thereby maximizing the attitude update rate. A fiducial marker system uses five infrared light-emitting diodes with four in the same plane and the fifth outside the plane. This noncoplanar design not only improves the attitude determination accuracy, but also provides an element of robustness via a fiducial marker fault diagnosis and process method in case one of the light-emitting diodes is obscured by the top balance mass or is faulty. The static experimental results show that the system can provide attitude accuracy of 0.06 deg and angular velocity accuracy of with an attitude update rate of at least 10 Hz. Finally, a yaw-axis maneuver is performed to demonstrate the system performance under dynamic conditions. The developed system could be reused to support testing for other scientific space missions, which represents a major added value of this work.

Pi-Tag: a fast image-space marker design based on projective invariants

Visual marker systems have become an ubiquitous tool to supply a reference frame onto otherwise uncontrolled scenes. Throughout the last decades, a wide range of different approaches have emerged, each with dif- ferent strengths and limitations. Some tags are optimized to reach a high accuracy in the recovered camera pose, others are based on designs that aim to maximizing the detection speed or minimizing the effect of occlusion on the detec- tion process. Most of them, however, employ a two-step pro- cedure where an initial homography estimation is used to translate the marker from the image plane to an orthonormal world, where it is validated and recognized. In this paper, we present a general purpose fiducial marker system that performs both steps directly in image-space. Specifically, by exploiting projective invariants such as collinearity and cross-ratios, we introduce a detection and recognition algo- rithm that is fast, accurate and moderately robust to occlu- sion. The overall performance of the system is evaluated in an extensive experimental section, where a comparison with a well-known baseline technique is presented. Additionally, several real-world applications are proposed, ranging from camera calibration to projector-based augmented reality.

SU-E-J-40: Lung Lesion Tracking Using Fixed-Spaced Non-Migrating Fiducial Markers in Robotic Radiosurgery.

With lung lesion treatment being a major indication of the use of the CyberKnife (CK) robotic radiosurgery system (Accuray Inc, Sunnyvale, CA), the ability for the CK's stereoscopic kV imaging system to accurately track implanted fiducial markers becomes vital in the accurate delivery of therapeutic radiation. This study examines a novel fixed-space fiducial marker delivery system which is capable of delivering two VISICOIL (IBA Dosimetry America, Memphis, TN) non-migrating fiducial markers simultaneously at a fixed spacing through a single 20-gauge needle. More specifically, presented herein is a preliminary study which tests the CK'streatment localization system's (TLS) ability to track markers of varying size and spacing, comparing subsequent stereoscopic kV imaging to DRR's generated during the planning stages. Three markers were placed in an XLT Lung Phantom (CIRS Inc, Norfolk, VA); two markers inserted along a diagonal line in a coronal plane, separated by biocompatible spacersof varying size, and a common third marker being placed in a non-varyinglocation in a coronal plane anterior to the marker pair. This third marker allows the calculation of rotational and translation corrections. Differentcombinations were scanned, planned, and simulated; 3.5mm- and 5.0mm- long markers, each 5mm in diameter, were separated by 15mm, 17mm, and 20mm spacers. The TLS system was able to track each of the aforementioned configurations with standard lung imaging parameters.Longer markers were not included in the study since earlier studies showed that without the natural deformation that would occur upon implantation, the length would induce false tracking. This is a necessary firststep in determining the minimum spacing with which the CK's TLS can track, a study which can now proceed with the use of phantom treatments elivered to orthogonally-overlapping radiochromic film, bisecting a tumor volume which is implanted with this fiducial marker system. Research supported in-part by IBA Dosimetry America.

ON THE DESIGN AND EVALUATION OF A PRECISE, ROBUST AND SCALABLE FIDUCIAL MARKER FRAMEWORK

In this paper we present an improved color-based planar fiducial marker system. Our framework provides precise and robust full 3D pose estimation of markers with superior accuracy when compared with many fiducial systems in the literature, while color information encoding enables using over 65 000 distinct markers. Unlike most color-based fiducial frameworks, which requires prior classification training and color calibration, ours can perform reliably under illumination changes, requiring but a rough white balance adjustment. Our methodology provides good detection performance even under poor illumination conditions which typically compromise other marker identification techniques, thus avoiding the evaluation of otherwise falsely identified markers. Several experiments are presented and carefully analyzed, in order to validate our system and demonstrate the significant improvement in estimation accuracy of both position and orientation over traditional techniques.

Designing Highly Reliable Fiducial Markers

Fiducial markers are artificial landmarks added to a scene to facilitate locating point correspondences between images, or between images and a known model. Reliable fiducials solve the interest point detection and matching problems when adding markers is convenient. The proper design of fiducials and the associated computer vision algorithms to detect them can enable accurate pose detection for applications ranging from augmented reality, input devices for HCI, to robot navigation. Marker systems typically have two stages, hypothesis generation from unique image features and verification/identification. A set of criteria for high robustness and practical use are identified and then optimized to produce the ARTag fiducial marker system. An edge-based method robust to lighting and partial occlusion is used for the hypothesis stage, and a reliable digital coding system is used for the identification and verification stage. Using these design criteria large gains in performance are achieved by ARTag over conventional ad hoc designs.