Multiple Fisheye Cameras Research Articles

FishSeg: 3D Fish Tracking Using Mask R-CNN in Large Ethohydraulic Flumes

To study the fish behavioral response to up- and downstream fish passage structures, live-fish tests are conducted in large flumes in various laboratories around the world. The use of multiple fisheye cameras to cover the full width and length of a flume, low color contrast between fish and flume bottom and non-uniform illumination leading to fish shadows, air bubbles wrongly identified as fish as well as fish being partially hidden behind each other are the main challenges for video-based fish tracking. This study improves an existing open-source fish tracking code to better address these issues by using a modified Mask Regional-Convolutional Neural Network (Mask R-CNN) as a tracking method. The developed workflow, FishSeg, consists of four parts: (1) stereo camera calibration, (2) background subtraction, (3) multi-fish tracking using Mask R-CNN, and (4) 3D conversion to flume coordinates. The Mask R-CNN model was trained and validated with datasets manually annotated from background subtracted videos from the live-fish tests. Brown trout and European eel were selected as target fish species to evaluate the performance of FishSeg with different types of body shapes and sizes. Comparison with the previous method illustrates that the tracks generated by FishSeg are about three times more continuous with higher accuracy. Furthermore, the code runs more stable since fish shadows and air bubbles are not misidentified as fish. The trout and eel models produced from FishSeg have mean Average Precisions (mAPs) of 0.837 and 0.876, respectively. Comparisons of mAPs with other R-CNN-based models show the reliability of FishSeg with a small training dataset. FishSeg is a ready-to-use open-source code for tracking any fish species with similar body shapes as trout and eel, and further fish shapes can be added with moderate effort. The generated fish tracks allow researchers to analyze the fish behavior in detail, even in large experimental facilities.

Toward Multicamera Systems for Minimally Invasive Surgery

Nowadays, many surgical operations for the diagnosis and treatment of diseases are performed using minimally invasive techniques. This trend has increased the demand for ergonomic medical devices to help surgeons safely execute medical practices by enhancing their experience, thus improving surgical outcomes. In this context, one of the most important challenges of minimally invasive surgery is the narrow field of view: Surgeons can only see the instrument's tip movement on a 2-D screen and cannot monitor the adjacent anatomical areas in three dimensions as in traditional open surgery. Hence, perforation of healthy tissues or undesired brushing may occur. To cope with these problems, we propose a novel surgical instrument, which embeds multiple fisheye cameras, accompanied by an image processing algorithm to visualize the adjacent area from different angles. This will allow surgeons to have a broader view around the instrument itself, hence preventing injuries caused by contact between the instrument body and the surrounding anatomical areas. The proposed system is versatile: It can be used for different surgical operations, and it can also be adapted for other applications dealing with narrow space exploration requiring a broad view.

Vehicle video surveillance system based on image fusion and parallel computing

SummaryAutonomous driving has gradually moved towards practical applications in recent years. It is particularly critical to provide reliable real‐time environmental information for autonomous driving systems. At present, vehicle video surveillance systems based on multi‐source video and target detection algorithms can effectively solve these problems. However, the previous vehicle video surveillance systems are often unable to balance the surveillance effect and the surveillance frame rate. Therefore, we will introduce a vehicle video surveillance system based on parallel computing and computer vision in this article. First, multiple fisheye cameras are used to collect surround‐view environmental information. Second, we will use a low‐light camera, infrared thermal imager, and millimeter‐wave radar to provide forward‐view environmental information at night. Correspondingly, we designed the surround‐view image fusion algorithm and the forward‐view image fusion algorithm based on parallel computing. At last, a monocular camera and detection algorithms are used to provide forward‐view detection results. In a word, this vehicle video surveillance system will benefit the practical application of autonomous driving.

Free viewpoint image generation system using fisheye cameras and a laser rangefinder for indoor robot teleoperation

In robot teleoperation, a lack of depth information often results in collisions between the robots and obstacles in its path or surroundings. To address this issue, free viewpoint images can greatly benefit the operators in terms of collision avoidance as the operators are able to view the robot’s surrounding from the images at arbitrary points, giving them a better depth information. In this paper, a novel free viewpoint image generation system is proposed. One approach to generate free viewpoint images is to use multiple cameras and Light Detection and Ranging (LiDAR). Instead of using the expensive LiDAR, this study utilizes a cost-effective laser rangefinder (LRF) and a characteristic of man-made environments. In other words, we install multiple fisheye cameras and an LRF on a robot. Free viewpoint images are generated under the assumption that walls are perpendicular to the floor. Furthermore, an easy calibration for estimating the poses of the multiple fisheye cameras, the LRF, and the robot model is proposed. Experimental results show that the proposed method can generate free viewpoint images using cameras and an LRF. Finally, the proposed method is primarily implemented using OpenGL Shading Language to utilize a graphics processing unit computation to achieve a real-time processing of the multiple high-resolution images. Supplementary videos and our source code are available at our project page (https://matsuren.github.io/fvp).

Sensor Calibration for Arbitrary Viewpoint Visualization System with Laser Radar and Multiple Fisheye Cameras

It is important to visualize a teleoperated robot and its surrounding environment to the operator. An arbitrary viewpoint visualization system generates images that simulates to be taken from external viewpoint with onboard sensors. It is necessary to calibrate the sensors for field utilization of the visualization system. In this research, a sensor calibration method for a laser radar and multiple fisheye cameras is proposed. The proposed method uses the shape model of machine and multiple planes in the surrounding environment to calibrate fisheye cameras and laser radar. The calibration experiments was conducted with the real machine. The obtained image by the calibration result of the proposed method has low discrepancy between laser radar mesh and fisheye camera texture.

VINS-MKF：A Tightly-Coupled Multi-Keyframe Visual-Inertial Odometry for Accurate and Robust State Estimation.

State estimation is crucial for robot autonomy, visual odometry (VO) has received significant attention in the robotics field because it can provide accurate state estimation. However, the accuracy and robustness of most existing VO methods are degraded in complex conditions, due to the limited field of view (FOV) of the utilized camera. In this paper, we present a novel tightly-coupled multi-keyframe visual-inertial odometry (called VINS-MKF), which can provide an accurate and robust state estimation for robots in an indoor environment. We first modify the monocular ORBSLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization and Mapping) to multiple fisheye cameras alongside an inertial measurement unit (IMU) to provide large FOV visual-inertial information. Then, a novel VO framework is proposed to ensure the efficiency of state estimation, by adopting a GPU (Graphics Processing Unit) based feature extraction method and parallelizing the feature extraction thread that is separated from the tracking thread with the mapping thread. Finally, a nonlinear optimization method is formulated for accurate state estimation, which is characterized as being multi-keyframe, tightly-coupled and visual-inertial. In addition, accurate initialization and a novel MultiCol-IMU camera model are coupled to further improve the performance of VINS-MKF. To the best of our knowledge, it’s the first tightly-coupled multi-keyframe visual-inertial odometry that joins measurements from multiple fisheye cameras and IMU. The performance of the VINS-MKF was validated by extensive experiments using home-made datasets, and it showed improved accuracy and robustness over the state-of-art VINS-Mono.

Reconstructing atmospheric cloud particles from multiple fisheye cameras

In this article we address the problem of accurately reconstructing the 3D position and short-term motion of atmospheric cloud using passive instruments. The proposed system is based on multiple fish-eye cameras that capture daylight sky images in a synchronous manner. After introducing the inherent difficulties of the task, we propose novel methods for improving robustness and accuracy in the given application scenario. We emphasize the effectiveness of a multi-view (in our example a trinocular) camera system over a stereo camera pair with respect to reliability and precision of both 3D position and motion reconstructions. During the evaluation of the methods we address limitations and possible effective parametrization of the system, including the positioning of the cameras.

Development of Informal Communication Environment Using Interactive Tiled Display Wall

In the early stage of planning or design, it is desired to support the informal communication between remote sites. In this study, informal communication environment between remote places was constructed by using the interactive tiled display wall.This system measures the user's gesture as well as captures the user's image by using the multiple fish-eye cameras. Then, the remote users can communicate with each other by using the touch interaction in the shared space. This system was applied to the remote communication and the effectiveness of this method in the informal communication was experimentally evaluated.