Scene Segmentation Research Articles

Perceptual Grouping During Binocular Rivalry in Mild Glaucoma.

PurposeThis study tested perceptual grouping during binocular rivalry to probe the strength of neural connectivity of the visual cortex involved in early visual processing in patients with mild glaucoma.MethodsSeventeen patients with mild glaucoma with no significant visual field defects and 14 healthy controls participated. Rivalry stimuli were 1.8°-diameter discs, containing horizontal or vertical sine-wave gratings, viewed dichoptically. To test the grouping, two spatially separated identical stimuli were presented eccentrically to the same or different eyes and to the same or different hemifields. The outcome measures were the time of exclusive dominance of the grouped percept (i.e., percept with synchronized orientations), the rivalry rate, and the epochs of exclusive dominance.ResultsFor both groups, the grouping occurred primarily for the matching orientations in the same eye/same hemifield (MO SE/SH) and for the matching orientations in the same eye/different hemifield (MO SE/DH) conditions. Time dominance of the grouped percept of the glaucoma group was similar to that of the control group in all conditions. The rivalry rates in the MO SE/SH and MO SE/DH conditions were significantly larger in the control group than in the glaucoma group. The epochs of exclusive dominance of the grouped percept in the MO SE/SH condition were a median of 48-ms longer for the control group, but a median of 116-ms shorter for the glaucoma group when compared to those in the MO SE/DH condition.ConclusionPatients with mild glaucoma show clear impairments in binocular rivalry while evidence for deficits in perceptual grouping could be inferred only indirectly. If these deficits truly exist, they may have implications for higher levels of visual processing, such as object recognition and scene segmentation, but these predictions remain to be tested in future studies.

Beyond Monocular Deraining: Parallel Stereo Deraining Network Via Semantic Prior

Rain is a common natural phenomenon. Taking images in the rain however often results in degraded quality of images, thus compromises the performance of many computer vision systems. Most existing de-rain algorithms use only one single input image and aim to recover a clean image. Few work has exploited stereo images. Moreover, even for single image based monocular deraining, many current methods fail to complete the task satisfactorily because they mostly rely on per pixel loss functions and ignore semantic information. In this paper, we present a Paired Rain Removal Network (PRRNet), which exploits both stereo images and semantic information. Specifically, we develop a Semantic-Aware Deraining Module (SADM) which solves both tasks of semantic segmentation and deraining of scenes, and a Semantic-Fusion Network (SFNet) and a View-Fusion Network (VFNet) which fuse semantic information and multi-view information respectively. In addition, we also introduce an Enhanced Paired Rain Removal Network (EPRRNet) which exploits semantic prior to remove rain streaks from stereo images. We first use a coarse deraining network to reduce the rain streaks on the input images, and then adopt a pre-trained semantic segmentation network to extract semantic features from the coarse derained image. Finally, a parallel stereo deraining network fuses semantic and multi-view information to restore finer results. We also propose new stereo based rainy datasets for benchmarking. Experiments on both monocular and the newly proposed stereo rainy datasets demonstrate that the proposed method achieves the state-of-the-art performance. https://github.com/HDCVLab/Stereo-Image-Deraining .

PyraPVConv: Efficient 3D Point Cloud Perception with Pyramid Voxel Convolution and Sharable Attention.

Designing efficient deep learning models for 3D point cloud perception is becoming a major research direction. Point-voxel convolution (PVConv) Liu et al. (2019) is a pioneering research work in this topic. However, since with quite a few layers of simple 3D convolutions and linear point-voxel feature fusion operations, it still has considerable room for improvement in performance. In this paper, we propose a novel pyramid point-voxel convolution (PyraPVConv) block with two key structural modifications to address the above issues. First, PyraPVConv uses a voxel pyramid module to fully extract voxel features in the manner of feature pyramid, such that sufficient voxel features can be obtained efficiently. Second, a sharable attention module is utilized to capture compatible features between multi-scale voxels in pyramid and point cloud for aggregation, as well as to reduce the complexity via structure sharing. Extensive results on three point cloud perception tasks, i.e., indoor scene segmentation, object part segmentation and 3D object detection, validate that the networks constructed by stacking PyraPVConv blocks are efficient in terms of both GPU memory consumption and computational complexity, and are superior to the state-of-the-art methods.

Multi-LiDAR Mapping for Scene Segmentation in Indoor Environments for Mobile Robots.

Nowadays, most mobile robot applications use two-dimensional LiDAR for indoor mapping, navigation, and low-level scene segmentation. However, single data type maps are not enough in a six degree of freedom world. Multi-LiDAR sensor fusion increments the capability of robots to map on different levels the surrounding environment. It exploits the benefits of several data types, counteracting the cons of each of the sensors. This research introduces several techniques to achieve mapping and navigation through indoor environments. First, a scan matching algorithm based on ICP with distance threshold association counter is used as a multi-objective-like fitness function. Then, with Harmony Search, results are optimized without any previous initial guess or odometry. A global map is then built during SLAM, reducing the accumulated error and demonstrating better results than solo odometry LiDAR matching. As a novelty, both algorithms are implemented in 2D and 3D mapping, overlapping the resulting maps to fuse geometrical information at different heights. Finally, a room segmentation procedure is proposed by analyzing this information, avoiding occlusions that appear in 2D maps, and proving the benefits by implementing a door recognition system. Experiments are conducted in both simulated and real scenarios, proving the performance of the proposed algorithms.

Data-centric multi-task surgical phase estimation with sparse scene segmentation

PurposeSurgical workflow estimation techniques aim to divide a surgical video into temporal segments based on predefined surgical actions or objectives, which can be of different granularity such as steps or phases. Potential applications range from real-time intra-operative feedback to automatic post-operative reports and analysis. A common approach in the literature for performing automatic surgical phase estimation is to decouple the problem into two stages: feature extraction from a single frame and temporal feature fusion. This approach is performed in two stages due to computational restrictions when processing large spatio-temporal sequences.MethodsThe majority of existing works focus on pushing the performance solely through temporal model development. Differently, we follow a data-centric approach and propose a training pipeline that enables models to maximise the usage of existing datasets, which are generally used in isolation. Specifically, we use dense phase annotations available in Cholec80, and sparse scene (i.e., instrument and anatomy) segmentation annotation available in CholecSeg8k in less than 5% of the overlapping frames. We propose a simple multi-task encoder that effectively fuses both streams, when available, based on their importance and jointly optimise them for performing accurate phase prediction.Results and conclusionWe show that with a small fraction of scene segmentation annotations, a relatively simple model can obtain comparable results than previous state-of-the-art and more complex architectures when evaluated in similar settings. We hope that this data-centric approach can encourage new research directions where data, and how to use it, plays an important role along with model development.

MFNet algorithm based on indoor scene segmentation

With the advancement of computer performance, deep learning is playing a vital role on hardware platforms. Indoor scene segmentation is a challenging deep learning task because indoor objects tend to obscure each other, and the dense layout increases the difficulty of segmentation. Still, current networks pursue accuracy improvement, sacrifice speed, and augment memory resource usage. To solve this problem, achieve a compromise between accuracy, speed, and model size. This paper proposes Multichannel Fusion Network (MFNet) for indoor scene segmentation, which mainly consists of Dense Residual Module(DRM) and Multi-scale Feature Extraction Module(MFEM). MFEM uses depthwise separable convolution to cut the number of parameters, matches different sizes of convolution kernels and dilation rates to achieve optimal receptive field; DRM fuses feature maps at several levels of resolution to optimize segmentation details. Experimental results on the NYU V2 dataset show that the proposed method achieves very competitive results compared with other advanced algorithms, with a segmentation speed of 38.47 fps, nearly twice that of Deeplab v3+, but only 1/5 of the number of parameters of Deeplab v3 + . Its segmentation results were close to those of advanced segmentation networks, making it beneficial for the real-time processing of images.

Paying attention for adjacent areas: Learning discriminative features for large-scale 3D scene segmentation

Despite recent improvements in analyzing large-scale 3D point clouds, several problems still exist: (a) segmentation models suffer from intra-class inconsistency and inter-class indistinction; (b) the existing methods ignore the inherent long-tailed class distribution of real-world 3D data. These problems result in unsatisfactory semantic segmentation predictions, especially in object adjacent areas. To handle these problems, this paper proposes a novel Adjacent areas Refinement Network (ARNet). Specifically, an Adjacent areas Refinement (AR) module is designed, which consists of two parallel attention blocks. Besides, our proposed attention blocks can process a large number of points (N∼105) with a slight increase in the computational complexity and time cost. Additionally, to deal with the inherent long-tailed class distribution in real-world 3D data, imbalance adjustment loss and occupancy regression loss are introduced. Based on this, the proposed network can handle the classification of both majority and minority classes, which is essential in distinguishing the ambiguous parts in large-scale 3D scenes. The proposed AR module and the loss functions can be easily integrated into the cutting-edge backbone networks, contributing to better performance in modeling semantic inter-dependencies and significantly improving the accuracy of the state-of-the-art semantic segmentation methods on indoor and outdoor scenes.

FPCC: Fast point cloud clustering-based instance segmentation for industrial bin-picking

Instance segmentation is an important pre-processing task in numerous real-world applications, such as robotics, autonomous vehicles, and human–computer interaction. Compared with the rapid development of deep learning for two-dimensional (2D) image tasks, deep learning-based instance segmentation of 3D point cloud still has a lot of room for development. In particular, distinguishing a large number of occluded objects of the same class is a highly challenging problem, which is seen in robotic bin-picking. In a usual bin-picking scene, many identical objects are stacked together and the model of the objects is known. Thus, the semantic information can be ignored; instead, the focus in the bin-picking is put on the segmentation of instances. Based on this task requirement, we propose a Fast Point Cloud Clustering (FPCC) for instance segmentation of industrial bin-picking scene. FPCC includes a network named FPCC-Net and a fast clustering algorithm. FPCC-Net extracts features of each point and infers geometric center points of each instance simultaneously. After that, the proposed clustering algorithm clusters the remaining points to the closest geometric center in feature embedding space. Experiments show that FPCC also surpasses the existing works in bin-picking scenes and is more computationally efficient. Our code and data are available at (https://github.com/xyjbaal/FPCC).

An Efficient Plane-Segmentation Method for Indoor Point Clouds Based on Countability of Saliency Directions

This paper proposes an efficient approach for the plane segmentation of indoor and corridor scenes. Specifically, the proposed method first uses voxels to pre-segment the scene and establishes the topological relationship between neighboring voxels. The voxel normal vectors are projected onto the surface of a Gaussian sphere based on the corresponding directions to achieve fast plane grouping using a variant of the K-means approach. To improve the segmentation integration, we propose releasing the points from the specified voxels and establishing second-order relationships between different primitives. We then introduce a global energy-optimization strategy that considers the unity and pairwise potentials while including high-order sequences to improve the over-segmentation problem. Three benchmark methods are introduced to evaluate the properties of the proposed approach by using the ISPRS benchmark datasets and self-collected in-house. The results of our experiments and the comparisons indicate that the proposed method can return reliable segmentation with precision over 72% even with the low-cost sensor, and provide the best performances in terms of the precision and recall rate compared to the benchmark methods.

DroneSegNet: Robust Aerial Semantic Segmentation for UAV-Based IoT Applications

Unmanned Aerial Vehicles (UAVs) are the promising “Flying IoT” devices of the future, which can be equipped with various sensors and cognitive capabilities to perform numerous tasks related to remote sensing, search and rescue operations, object tracking, segmentation of roads and buildings, surveillance, etc. However, these AI-driven tasks require heavy computation and may lead to suboptimal performance with embedded processors on a power-constrained battery-operated drone. This work proposes a novel deep learning approach for performing robust semantic segmentation of aerial scenes captured by UAVs. In our setup, the power-constrained drone is used only for data collection, while the computationally intensive tasks are offloaded to a GPU cloud server. Our architecture performs robust semantic segmentation by learning the segmentation maps from jointly utilizing of aerial scenes along with the respective “elevation maps” in a semi-supervised approach. We propose a three-tier deep learning architecture, wherein the first module aims at preliminary feature extraction from aerial scenes using a backbone feature extractor. The second module captures the spatial dependency between the aerial scenes and their respective elevation maps to obtain better semantic information, which is achieved by a bi-directional LSTM. The third module is aimed at enhancing the performance of semantic segmentation through a semi-supervised approach with an encoder to generate segmentation maps and a decoder to reconstruct feature maps. This semi-supervised feature learning ensures robust extraction along with scalability. The proposed architecture was validated on real-world aerial datasets and achieves state-of-the-art results for aerial image segmentation.

Bird’s-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird’s-Eye-View (BEV) maps have emerged as one of the most powerful representations for scene understanding due to their ability to provide rich spatial context while being easy to interpret and process. Such maps have found use in many real-world tasks that extensively rely on accurate scene segmentation as well as object instance identification in the BEV space for their operation. However, existing segmentation algorithms only predict the semantics in the BEV space, which limits their use in applications where the notion of object instances is also critical. In this work, we present the first BEV panoptic segmentation approach for directly predicting dense panoptic segmentation maps in the BEV, given a single monocular image in the frontal view (FV). Our architecture follows the top-down paradigm and incorporates a novel dense transformer module consisting of two distinct transformers that learn to independently map vertical and flat regions in the input image from the FV to the BEV. Additionally, we derive a mathematical formulation for the sensitivity of the FV-BEV transformation which allows us to intelligently weight pixels in the BEV space to account for the varying descriptiveness across the FV image. Extensive evaluations on the KITTI-360 and nuScenes datasets demonstrate that our approach exceeds the state-of-the-art in the PQ metric by 3.61 pp and 4.93 pp respectively.

Complex Scene Segmentation Network Based on Multi-scale Encoding-decoding Architecture

Abstract With the progress of artificial intelligence, the study of scene segmentation for complex scene understanding is of great significance. Due to the large number of activities, there are many target categories, large scale changes, many mutual occlusions, difficult target recognition, and large data labeling costs. In order to achieve accurate understanding of the complex scene, this paper proposes to add a scale adaptive feature module on the basis of Encode-Decode, so that the network can make good use of the features and context information of each level to effectively adapt to changes in target size. At the same time, we use the scale size function to weight encode different levels of features, which improves the prediction accuracy of pixels in the intersection area of each class. Experiments conducted on Cityscapes, Put_campus and PASCAL VOC 2012 datasets show that the method in this article is improved by about 1% compared with the three segmentation algorithms such as FCN, PSPNet, and Deeplabv3 +.

Improving Semantic Segmentation of Urban Scenes for Self-Driving Cars with Synthetic Images.

Semantic segmentation of an incoming visual stream from cameras is an essential part of the perception system of self-driving cars. State-of-the-art results in semantic segmentation have been achieved with deep neural networks (DNNs), yet training them requires large datasets, which are difficult and costly to acquire and time-consuming to label. A viable alternative to training DNNs solely on real-world datasets is to augment them with synthetic images, which can be easily modified and generated in large numbers. In the present study, we aim at improving the accuracy of semantic segmentation of urban scenes by augmenting the Cityscapes real-world dataset with synthetic images generated with the open-source driving simulator CARLA (Car Learning to Act). Augmentation with synthetic images with a low degree of photorealism from the MICC-SRI (Media Integration and Communication Center–Semantic Road Inpainting) dataset does not result in the improvement of the accuracy of semantic segmentation, yet both MobileNetV2 and Xception DNNs used in the present study demonstrate a better accuracy after training on the custom-made CCM (Cityscapes-CARLA Mixed) dataset, which contains both real-world Cityscapes images and high-resolution synthetic images generated with CARLA, than after training only on the real-world Cityscapes images. However, the accuracy of semantic segmentation does not improve proportionally to the amount of the synthetic data used for augmentation, which indicates that augmentation with a larger amount of synthetic data is not always better.

A Few-Shot Learning-Based Retinal Vessel Segmentation Method for Assisting in the Central Serous Chorioretinopathy Laser Surgery.

BackgroundThe location of retinal vessels is an important prerequisite for Central Serous Chorioretinopathy (CSC) Laser Surgery, which does not only assist the ophthalmologist in marking the location of the leakage point (LP) on the fundus color image but also avoids the damage of the laser spot to the vessel tissue, as well as the low efficiency of the surgery caused by the absorption of laser energy by retinal vessels. In acquiring an excellent intra- and cross-domain adaptability, the existing deep learning (DL)-based vessel segmentation scheme must be driven by big data, which makes the densely annotated work tedious and costly.MethodsThis paper aims to explore a new vessel segmentation method with a few samples and annotations to alleviate the above problems. Firstly, a key solution is presented to transform the vessel segmentation scene into the few-shot learning task, which lays a foundation for the vessel segmentation task with a few samples and annotations. Then, we improve the existing few-shot learning framework as our baseline model to adapt to the vessel segmentation scenario. Next, the baseline model is upgraded from the following three aspects: (1) A multi-scale class prototype extraction technique is designed to obtain more sufficient vessel features for better utilizing the information from the support images; (2) The multi-scale vessel features of the query images, inferred by the support image class prototype information, are gradually fused to provide more effective guidance for the vessel extraction tasks; and (3) A multi-scale attention module is proposed to promote the consideration of the global information in the upgraded model to assist vessel localization. Concurrently, the integrated framework is further conceived to appropriately alleviate the low performance of a single model in the cross-domain vessel segmentation scene, enabling to boost the domain adaptabilities of both the baseline and the upgraded models.ResultsExtensive experiments showed that the upgraded operation could further improve the performance of vessel segmentation significantly. Compared with the listed methods, both the baseline and the upgraded models achieved competitive results on the three public retinal image datasets (i.e., CHASE_DB, DRIVE, and STARE). In the practical application of private CSC datasets, the integrated scheme partially enhanced the domain adaptabilities of the two proposed models.

Value of Temporal Dynamics Information in Driving Scene Segmentation

Semantic scene segmentation has primarily been addressed by forming high-level visual representations of single images. The problem of semantic segmentation in dynamic scenes has begun to receive attention with the video object segmentation and tracking problem. While there has been some recent work attempt to use deep learning models on the video level, what is not known is how the temporal dynamics information is contributing to the full scene segmentation. Moreover, most existing datasets only provide full scene annotation on non-consecutive images to ensure the variability of scenes, making it even harder to explore novel methods on video-level modeling. To address the above issues, our work takes steps to explore the behavior of modern spatiotemporal modeling approaches by: 1) constructing the MIT DriveSeg dataset, a large-scale video driving scene segmentation dataset, densely annotated for pixel-level semantic classes with 5000 consecutive video frames, and 2) proposing a joint-learning framework that reveals the contribution of temporal dynamics information in regard to different semantic classes in the driving scene. This work is intended to help assess current methods and support further exploration of the value of temporal dynamics information in video-level scene segmentation.

High-performance panoramic annular lens design for real-time semantic segmentation on aerial imagery

As the visual perception window of the drone system, the lens provides great help for obtaining visual information, detection, and recognition. However, traditional lenses carried on drones cannot have characteristics of a large field of view (FoV), small size, and low weight at the same time. To meet the above requirements, we propose a panoramic annular lens (PAL) system with 4K high resolution, a large FoV of (30 deg to 100 deg) × 360 deg, an angular resolution of 12.2 mrad of aerial perspective, and great imaging performance. We equip a drone system with our designed PAL to collect panoramic image data at an altitude of 100 m from the track and field and obtain the first drone-perspective panoramic scene segmentation dataset Aerial-PASS, with annotated labels of track and field. We design an efficient deep architecture for aerial scene segmentation. Trained on Aerial-PASS, the yielded model accurately segments aerial images. Compared with the ERF-PAPNet and SwiftNet semantic segmentation networks, the network we adopted has higher recognition accuracy with the mean IoU greater than 86.30%, which provides an important reference for the drone system to monitor and identify specific targets in real-time in a large FoV.

Indoor Target Detection and Pose Estimation Based on Point Cloud

The indoor environment faced by service robots is complex, with plenty of objects and mutual occlusion, traditional algorithms cannot detect the target and estimate its pose accurately. We propose a set of target detection and pose estimation algorithm based on point cloud templates to solve the above problems. Through the segmentation and clustering of scenes, the application of the algorithm in complex scenes is realized. Based on the key points to estimate the pose of the target, we achieve the adaptation to mutual occlusion. Besides, we propose an improved RANSAC algorithm, which maintains the accuracy of the original algorithm while greatly improving the speed. Finally, we test the algorithm through the open source model library, simulated and actual scenarios, compared with the SAC-IA algorithm, our algorithm improves the time by 89%, and in the application of complex scenes, the error of the pose estimation is at the millimeter level.

LFPNet: Lightweight network on real point sets for fruit classification and segmentation

• A lightweight fruit phenotypic processing network is proposed to solve the complex calculation of the fruit features recognition in real scenes. • By combining 3D-SPP and global feature feedback, a novel fruit segmentation network is designed to improve the segmentation integrity. • Based on preprocessing and 3D STN, LFPNet is proved to be generalization in directly processing the disordered fruit scenes without ideal labeled. • Our LFPNet has only 1.17 M parameters and improves 2.15% on mIoU, 3.175% on OA, 3.45% on mAcc. 3D point cloud reconstruction, as the key technology to obtain high-throughput fruit phenotypic data, has solved the problems caused by complex environments, high fruit similarity, and the lack of public datasets suitable for fruit characterization. However, in the process of identifying and segmenting fruit data from point cloud, the existing network architectures lead to problems such as classification error, incomplete segmentation and low efficiency. In this paper, we introduce LFPNet, a novel and efficient lightweight neural network that directly consumes fruit point clouds in the real scene. Our network mainly has the following three advantages: 1) The introduction of voxel-filter based down-sampling preprocessing can help to avoid classification error caused by invalid noise interference. 2) A 3D STN is designed to solve the lack of spatial invariance in convolutional neural network (CNN) when calculating and analyzing fruit point clouds. 3) By introducing spatial pyramid pooling and combining local and global features, a fruit segmentation network is built to improve the integrity of segmentation in fruit scenes. Experimental results show that our LFPNet performs as well as or better than most of its peers in terms of classification accuracy and segmentation integrity.

A VR-based CAD 3D Modeling System for Smart Home Products

There is still ample room for further optimization of the application effect of virtual reality (VR) in smart products, and further expansion of the reconstruction research of smart home products. Hence, the related algorithms of spatial reconstruction and three-dimensional (3D) Computer-Aided Design (CAD) item reconstruction is proposed based on the relevant concepts of CAD and convolutional neural network. The algorithms aim to construct the virtual application scene and the simulation state of product usage by VR technology, and stimulate customers’ purchase decision. Through the experiments, it is found that the Fully Convolutional Network optimized by Visual Geometry Group Network can effectively realize scene reconstruction and item reconstruction. The average overall recognition accuracy of scene segmentation is 91.3%, and the mean Intersection over Union is 63.4%, showing a better recognition effect than other similar algorithms. Moreover, in the detection of indoor items, the maximum detection accuracy of the MaskR-CNN algorithm is 0.89, and the detection effect is also better than that of other algorithms. The experiments achieve a good spatial reconstruction structure, with the average voxel Prec rates in unsheltered space and sheltered space of 94.4% and 55.6%, and the average voxel Recall of 92.2% and 36.5%, respectively. The experimental results indicate an excellent spatial reconstruction effect. This study can provide a more scientific reference for subsequent VR system research.

CARL-D: A vision benchmark suite and large scale dataset for vehicle detection and scene segmentation

Vision-based object detection and scene understanding are becoming key features of environment perception and autonomous driving. In the past couple of years, numerous large-scale datasets for visual object detection and semantic understanding have been released which has enormously benefited the environment perception in self-driving cars. However, these datasets like Kitti and Cityscapes only focus on well organized and urban road traffic scenarios of European countries while ignoring the dense and unpattern traffic conditions of sub-continent countries like Pakistan, India, Bangladesh, and Sri Lanka. Consequently, the environment perception system developed on these datasets cannot efficiently assist self-driving cars in traffic scenarios of sub-continent countries. To this end, we present CARL-D, a large-scale dataset and benchmark suite for develop 2D object detection and instance/pixel-level segmentation methods for self-driving cars. CARL-D comprises large-scale stereo vision-based driving videos captured from more than 100 cities of Pakistan, including motorways, dense and unpattern traffic scenarios of urban, rural, and hilly areas. As a benchmark selection, 15,000 suitable images are labeled for 2D object detection and recognition. Whereas semantic segmentation benchmark contains 2500 images with pixel-level high-quality fine annotations and 5000 coarse-annotated images which could help in enabling the deep neural networks to leverage the weakly labeled data. Alongside the dataset, we also present transfer learning-based 2D vehicle detection and scene segmentation methods to evaluate the performance of existing state-of-the-art deep neural networks on our dataset. Lastly, an extensive experimental evaluation along with the comparative study has been carried out which demonstrates the upper edge of our dataset in terms of interclass-diversity, scene-variability, and annotations richness. The proposed benchmark suite is available at https://carl-dataset.github.io/index/.