Mobile Device Camera Research Articles

Automatic detection of the parasite Trypanosoma cruzi in blood smears using a machine learning approach applied to mobile phone images.

Chagas disease is a life-threatening illness caused by the parasite Trypanosoma cruzi. The diagnosis of the acute form of the disease is performed by trained microscopists who detect parasites in blood smear samples. Since this method requires a dedicated high-resolution camera system attached to the microscope, the diagnostic method is more expensive and often prohibitive for low-income settings. Here, we present a machine learning approach based on a random forest (RF) algorithm for the detection and counting of T. cruzi trypomastigotes in mobile phone images. We analyzed micrographs of blood smear samples that were acquired using a mobile device camera capable of capturing images in a resolution of 12 megapixels. We extracted a set of features that describe morphometric parameters (geometry and curvature), as well as color, and texture measurements of 1,314 parasites. The features were divided into train and test sets (4:1) and classified using the RF algorithm. The values of precision, sensitivity, and area under the receiver operating characteristic (ROC) curve of the proposed method were 87.6%, 90.5%, and 0.942, respectively. Automating image analysis acquired with a mobile device is a viable alternative for reducing costs and gaining efficiency in the use of the optical microscope.

Deep Learning Methods for Accurate Skin Cancer Recognition and Mobile Application

Although many efforts have been made through past years, skin cancer recognition from medical images is still an active area of research aiming at more accurate results. Many efforts have been made in recent years based on deep learning neural networks. Only a few, however, are based on a single deep learning model and targeted to create a mobile application. Contributing to both efforts, first we present a summary of the required medical knowledge on skin cancer, followed by an extensive summary of the most recent related works. Afterwards, we present 11 CNN (convolutional neural network) candidate single architectures. We train and test those 11 CNN architectures, using the HAM10000 dataset, concerning seven skin lesion classes. To face the imbalance problem and the high similarity between images of some skin lesions, we apply data augmentation (during training), transfer learning and fine-tuning. From the 11 CNN architecture configurations, DenseNet169 produced the best results. It achieved an accuracy of 92.25%, a recall (sensitivity) of 93.59% and an F1-score of 93.27%, which outperforms existing state-of-the-art efforts. We used a light version of DenseNet169 in constructing a mobile android application, which was mapped as a two-class model (benign or malignant). A picture is taken via the mobile device camera, and after manual cropping, it is classified into benign or malignant type. The application can also inform the user about the allowed sun exposition time based on the current UV radiation degree, the phototype of the user’s skin and the degree of the used sunscreen. In conclusion, we achieved state-of-the-art results in skin cancer recognition based on a single, relatively light deep learning model, which we also used in a mobile application.

96.5 Using Computer Vision-Based Algorithms Trained on Mobile-Device Camera Images for Monitoring Burn Wound Healing

IntroductionThe appropriate characterization of burn depth and healing is paramount. Unfortunately, the accuracy of approximating thermal injury depth among all physicians is poor. While tools to improve detection accuracy, including laser doppler imaging and laser speckle imaging exist, these technologies are expensive and limited to specialized burn referral centres. They also do not provide an easy means for quantitative, interval tracking of burn healing. Considering these limitations, the application of artificial intelligence has garnered significant interest. We herein present the use of three novel machine learning and computer vision-based algorithms to track burn wound healing.MethodsConvolutional neural network (CNN) models, were trained on 1800 2D color burn images, to classify them into four burn severities. These CNNs were used to develop saliency algorithms that identify the highest “attention” pixels used to recognize burns. Image-based algorithms that count these attention pixels of the CNN, count pixels representing red granulation of burns, and measure burns, were also developed. As proof-of-concept, we tracked the healing of a localized burn on a 25-year-old female patient. The patient suffered a scald on the dorsum of the foot, resulting in a deep partial-thickness burn. Opting out of surgical intervention, the patient visited the hospital over a 6-week period for treatment with non-adhesive dressings and silver nitrate. High-resolution images of the burn, with and without a fiducial marker, were captured with a smartphone camera every 7-days. Images were taken under institutional lighting and used as algorithmic inputs.ResultsData analyses indicate that the healing of the open-wound area was accurately measured in millimetres (+/- 1.7 mm error) using a fiducial marker (18.3 mm diameter). The open-wound area shrank consistently from week 1 to week 6 seen in (Figure 1. a-b). The normalized, 2D colour images, where the “red” pixel value was counted (Figure 1. a-b), confirms the reduction of the red granulation in the wound. The saliency algorithm also measured a percentage reduction in the machine learning model’s total attention pixels over the 6-week period (Figure 1. c-d). This suggests that the model was less discerning of the healing burn wound over time, suggesting burn healing, which was also clinically validated.

Neural Network Design using a Virtual Reality Platform

The evolution of Deep Learning (DL), a subset of machine learning, has made their use very effective in many artificial intelligence(AI) fields. In parallel Virtual Reality is going wide in many applications thanks to the proliferation of cameras in mobile devices and improved processing efficiency. Data visualization in deep learning is a fundamental element for which it can benefit from the advantages offered by the visualization of the VR for the development of the models. In addition, the researchers can widely use the editing of images and videos in the machine learning process to design a convolutional network suitable for image recognition. In this study, we want to demonstrate the usefulness of this approach in collecting data within virtual reality to train and optimize a convolutional neural network used to recognize human activities (HAR).

Image quality evaluation of video conferencing solutions with realistic laboratory scenes

Video conferencing has become extremely relevant in the world in the latest years. Traditional image and video quality evaluation techniques prove insufficient to properly assess the quality of these systems, since they often include special processing pipelines, for example, to improve face rendering. Our team proposes a suite of equipment, laboratory scenes and measurements that include realistic mannequins to simulate a more true-to-life scene, while still being able to reliably measure image quality in terms of exposure, dynamic range, color and skin tone rendering, focus, texture, and noise. These metrics are used to evaluate and compare three categories of cameras for video conference that are available on the market: external webcams, laptop integrated webcams and selfie cameras of mobile devices. Our results showed that external webcams provide a real image quality advantage over most built-in webcams in laptops but cannot match the superior image quality of tablets and smartphones selfie cameras. Our results are consistent with perceptual evaluation and allow for an objective comparison of very different systems.

Human Monkeypox Classification from Skin Lesion Images with Deep Pre-trained Network using Mobile Application.

Recently, human monkeypox outbreaks have been reported in many countries. According to the reports and studies, quick determination and isolation of infected people are essential to reduce the spread rate. This study presents an Android mobile application that uses deep learning to assist this situation. The application has been developed with Android Studio using Java programming language and Android SDK 12. Video images gathered through the mobile device’s camera are dispatched to a deep convolutional neural network that runs on the same device. Camera2 API of the Android platform has been used for camera access and operations. The network then classifies images as positive or negative for monkeypox detection. The network’s training has been carried out using skin lesion images of monkeypox-infected people and other skin lesion images. For this purpose, a publicly available dataset and a deep transfer learning approach have been used. All training and testing steps have been applied on Matlab using different pre-trained networks. Then, the network that has the best accuracy has been recreated and trained using TensorFlow. The TensorFlow model has been adapted to mobile devices by converting to the TensorFlow Lite model. The TensorFlow Lite model has been then embedded into the mobile application together with the TensorFlow Lite library for monkeypox detection. The application has been run on three devices successfully. During the run-time, the inference times have been gathered. 197 ms, 91 ms, and 138 ms average inference times have been observed. The presented system allows people with body lesions to quickly make a preliminary diagnosis. Thus, monkeypox-infected people can be encouraged to act rapidly to see an expert for a definitive diagnosis. According to the test results, the system can classify the images with 91.11% accuracy. In addition, the proposed mobile application can be trained for the preliminary diagnosis of other skin diseases.

A Method of Image Quality Assessment for Text Recognition on Camera-Captured and Projectively Distorted Documents

In this paper, we consider the problem of identity document recognition in images captured with a mobile device camera. A high level of projective distortion leads to poor quality of the restored text images and, hence, to unreliable recognition results. We propose a novel, theoretically based method for estimating the projective distortion level at a restored image point. On this basis, we suggest a new method of binary quality estimation of projectively restored field images. The method analyzes the projective homography only and does not depend on the image size. The text font and height of an evaluated field are assumed to be predefined in the document template. This information is used to estimate the maximum level of distortion acceptable for recognition. The method was tested on a dataset of synthetically distorted field images. Synthetic images were created based on document template images from the publicly available dataset MIDV-2019. In the experiments, the method shows stable predictive values for different strings of one font and height. When used as a pre-recognition rejection method, it demonstrates a positive predictive value of 86.7% and a negative predictive value of 64.1% on the synthetic dataset. A comparison with other geometric quality assessment methods shows the superiority of our approach.

SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting

Recently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device cameras, the resolution of image and video sources available to users via the cloud or locally is increasing. For high-resolution images, the common inpainting methods simply upsample the inpainted result of the shrinked image to yield a blurry result. In recent years, there is an urgent need to reconstruct the missing high-frequency information in high-resolution images and generate sharp texture details. Hence, we propose a general deep learning framework for high-resolution image inpainting, which first hallucinates a semantically continuous blurred result using low-resolution inpainting and suppresses computational overhead. Then the sharp high-frequency details with original resolution are reconstructed using super-resolution refinement. Experimentally, our method achieves inspiring inpainting quality on 2K and 4K resolution images, ahead of the state-of-the-art high-resolution inpainting technique. This framework is expected to be popularized for high-resolution image editing tasks on personal computers and mobile devices in the future.

Three-dimensional-printed marker-based augmented reality neuronavigation: a new neuronavigation technique.

The aim of this study was to assess the precision and feasibility of 3D-printed marker-based augmented reality (AR) neurosurgical navigation and its use intraoperatively compared with optical tracking neuronavigation systems (OTNSs). Three-dimensional-printed markers for CT and MRI and intraoperative use were applied with mobile devices using an AR light detection and ranging (LIDAR) camera. The 3D segmentations of intracranial tumors were created with CT and MR images, and preoperative registration of the marker and pathology was performed. A patient-specific, surgeon-facilitated mobile application was developed, and a mobile device camera was used for neuronavigation with high accuracy, ease, and cost-effectiveness. After accuracy values were preliminarily assessed, this technique was used intraoperatively in 8 patients. The mobile device LIDAR camera was found to successfully overlay images of virtual tumor segmentations according to the position of a 3D-printed marker. The targeting error that was measured ranged from 0.5 to 3.5 mm (mean 1.70 ± 1.02 mm, median 1.58 mm). The mean preoperative preparation time was 35.7 ± 5.56 minutes, which is longer than that for routine OTNSs, but the amount of time required for preoperative registration and the placement of the intraoperative marker was very brief compared with other neurosurgical navigation systems (mean 1.02 ± 0.3 minutes). The 3D-printed marker-based AR neuronavigation system was a clinically feasible, highly precise, low-cost, and easy-to-use navigation technique. Three-dimensional segmentation of intracranial tumors was targeted on the brain and was clearly visualized from the skin incision to the end of surgery.

Real-time Hand Gesture Extraction Using Python Programming Language Facilities

Hand gesture recognition is one of communication in which used bodily behavior to transmit several messages. This paper aims to detect hand gestures with the mobile device camera and create a customize dataset that used in deep learning model training to recognize hand gestures. The real-time approach was used for all these objectives: the first step is hand area detection; the second step is hand area storing in a dataset form to use in the future for model training. A framework for human contact was put in place by studying pictures recorded by the camera. It was converted the RGB color space image to the greyscale, the blurring method is used for object noise removing efficaciously. To highlight the edges and curves of the hand, the thresholding method is used. And subtraction of complex background is applied to detect moving objects from a static camera. The objectives of the paper were reliable and favorable which helps deaf and dumb people interact with the environment through the sign language fully approved to extract hand movements. Python language as a programming manner to discover hand gestures. This work has an efficient hand gesture detection process to address the problem of framing from real-time video.

The effect of mobile camera selection on the capacity to predict water turbidity.

Recently, cameras of mobile have phones emerged as an alternative for quantifying water turbidity. Most of these studies lack a strategy to determine the water turbidity for new samples, focusing mainly on one particular device. Nevertheless, widespread use of these approaches requires a predictive capacity on out-of-the-sample images acquired in devices of different capabilities. We studied the influence of mobile device camera sensors on the predictive performance of water turbidity for non-previously observed turbid images. For this, a reference database with turbid images acquired for different mobile devices was constructed. A machine learning method based on image quality measures and linear classifiers (least squares and LASSO) was proposed to perform predictions on each mobile device. Relative accuracy and precision were evaluated. Results suggest that these approaches may provide accurate predictions reaching most than 80% of relative accuracy with high test-retest reliability (> 0.99). Nevertheless, our results also indicate that the predictive performance levels dropped in low capacity quality sensors. Therefore, despite the high performance that can be reached using these approaches, widespread use on multiple mobile devices may require further development of low-quality sensors and a better understanding of their operative ranges.

The UTrack framework for segmenting and measuring dermatological ulcers through telemedicine

Chronic dermatological ulcers cause great discomfort to patients, and while monitoring the size of wounds over time provides significant clues about the healing evolution and the clinical condition of patients, the lack of practical applications in existing studies impairs users’ access to appropriate treatment and diagnosis methods. We propose the UTrack framework to help with the acquisition of photos, the segmentation and measurement of wounds, the storage of photos and symptoms, and the visualization of the evolution of ulcer healing. UTrack-App is a mobile app for the framework, which processes images taken by standard mobile device cameras without specialized equipment and stores all data locally. The user manually delineates the regions of the wound and the measurement object, and the tool uses the proposed UTrack-Seg segmentation method to segment them. UTrack-App also allows users to manually input a unit of measurement (centimeter or inch) in the image to improve the wound area estimation. Experiments show that UTrack-Seg outperforms its state-of-the-art competitors in ulcer segmentation tasks, improving F-Measure by up to 82.5% when compared to superpixel-based approaches and up to 19% when compared to Deep Learning ones. The method is unsupervised, and it semi-automatically segments real-world images with 0.9 of F-Measure, on average. The automatic measurement outperformed the manual process in three out of five different rulers. UTrack-App takes at most 30 s to perform all evaluation steps over high-resolution images, thus being well-suited to analyze ulcers using standard mobile devices.

Proposing a Recognition System of Gestures Using MobilenetV2 Combining Single Shot Detector Network for Smart-Home Applications

The paper proposes a system for identifying gestures and actions in smart homes. The proposed method is based on MobilenetV2 feature extraction combining with single shot detector (SSD) network. We used eleven types of gestures of walking, sitting down, falling back, wearing shoes, waving hands, falling down, smoking, baby crawling, standing up, reading, and typing for recognizing the gestures. In this system, the data are captured from the camera of mobile devices that are used to detect the object. The results are obtained objects on the frame by a bounding box. The results show that the system meets the requirements with an accuracy of over 90% that is suitable for real application.

Weighted combination of per-frame recognition results for text recognition in a video stream

The scope of uses of automated document recognition has extended and as a result, recognition techniques that do not require specialized equipment have become more relevant. Among such techniques, document recognition using mobile devices is of interest. However, it is not always possible to ensure controlled capturing conditions and, consequentially, high quality of input images. Unlike specialized scanners, mobile cameras allow using a video stream as an input, thus obtaining several images of the recognized object, captured with various characteristics. In this case, a problem of combining the information from multiple input frames arises. In this paper, we propose a weighing model for the process of combining the per-frame recognition results, two approaches to the weighted combination of the text recognition results, and two weighing criteria. The effectiveness of the proposed approaches is tested using datasets of identity documents captured with a mobile device camera in different conditions, including perspective distortion of the document image and low lighting conditions. The experimental results show that the weighting combination can improve the text recognition result quality in the video stream, and the per-character weighting method with input image focus estimation as a base criterion allows one to achieve the best results on the datasets analyzed.

Algorithm for choosing the best frame in a video stream in the task of identity document recognition

During the process of document recognition in a video stream using a mobile device camera, the image quality of the document varies greatly from frame to frame. Sometimes recognition system is required not only to recognize all the specified attributes of the document, but also to select final document image of the best quality. This is necessary, for example, for archiving or providing various services; in some countries it can be required by law. In this case, recognition system needs to assess the quality of frames in the video stream and choose the “best” frame. In this paper we considered the solution to such a problem where the “best” frame means the presence of all specified attributes in a readable form in the document image. The method was set up on a private dataset, and then tested on documents from the open MIDV-2019 dataset. A practically applicable result was obtained for use in recognition systems.

Computer Vision Analysis for Quantification of Autism Risk Behaviors.

Observational behavior analysis plays a key role for the discovery and evaluation of risk markers for many neurodevelopmental disorders. Research on autism spectrum disorder (ASD) suggests that behavioral risk markers can be observed at 12 months of age or earlier, with diagnosis possible at 18 months. To date, these studies and evaluations involving observational analysis tend to rely heavily on clinical practitioners and specialists who have undergone intensive training to be able to reliably administer carefully designed behavioural-eliciting tasks, code the resulting behaviors, and interpret such behaviors. These methods are therefore extremely expensive, time-intensive, and are not easily scalable for large population or longitudinal observational analysis. We developed a self-contained, closed-loop, mobile application with movie stimuli designed to engage the child's attention and elicit specific behavioral and social responses, which are recorded with a mobile device camera and then analyzed via computer vision algorithms. Here, in addition to presenting this paradigm, we validate the system to measure engagement, name-call responses, and emotional responses of toddlers with and without ASD who were presented with the application. Additionally, we show examples of how the proposed framework can further risk marker research with fine-grained quantification of behaviors. The results suggest these objective and automatic methods can be considered to aid behavioral analysis, and can be suited for objective automatic analysis for future studies.

Abstract 17060: Machine Learning and Video Recognition for Automated Detection of Fluid Status in Heart Failure Patients

Introduction: There is a need for innovative solutions to better screen and diagnose the 7 million patients with chronic heart failure. A key component of assessing these patients is monitoring fluid status by evaluating for the presence and height of jugular venous distension (JVD). We hypothesize that video analysis of a patient’s neck using machine learning algorithms and image recognition can identify the amount of JVD. We propose the use of high fidelity video recordings taken using a mobile device camera to determine the presence or absence of JVD, which we will use to develop a point of care testing tool for early detection of acute exacerbation of heart failure. Methods: In this feasibility study, patients in the Duke cardiac catheterization lab undergoing right heart catheterization were enrolled. RGB and infrared videos were captured of the patient’s neck to detect JVD and correlated with right atrial pressure on the heart catheterization. We designed an adaptive filter based on biological priors that enhances spatially consistent frequency anomalies and detects jugular vein distention, with implementation done on Python. Results: We captured and analyzed footage for six patients using our model. Four of these six patients shared a similar strong signal outliner within the frequency band of 95bpm – 200bpm when using a conservative threshold, indicating the presence of JVD. We did not use statistical analysis given the small nature of our cohort, but in those we detected a positive JVD signal the RA mean was 20.25 mmHg and PCWP mean was 24.3 mmHg. Conclusions: We have demonstrated the ability to evaluate for JVD via infrared video and found a relationship with RHC values. Our project is innovative because it uses video recognition and allows for novel patient interactions using a non-invasive screening technique for heart failure. This tool can become a non-invasive standard to both screen for and help manage heart failure patients.

An attention recurrent model for human cooperation detection

User cooperative behaviour is mandatory and valuable to warranty data acquisition quality in forensic biometrics. In the present paper, we consider human cooperative behaviour in front of wearable security cameras. Moreover, we propose a human cooperation detection pipeline based on deep learning. Recently, recurrent neural networks (RNN) have shown remarkable performance on several tasks such as image captioning, video analysis, or natural language processing. Our proposal describes an RNN architecture with the aim at detecting whether a human is exhibiting an adversarial behaviour by trying to avoid the camera. This data is obtained by analysing the noise patterns of human movement. More specifically, we are not only providing an extensive analysis on the proposed pipeline considering different configurations and a wide variety of RNN types, but also an ensemble of the generated models to outperform each single model. The experiment has been carried out using videos captured from a mobile device camera (GOTCHA Dataset) and the obtained results have demonstrated the robustness of the proposed method.

Patients' Experiences and Attitudes of Using a Secure Mobile Phone App for Medical Photography: Qualitative Survey Study.

BackgroundPoint-of-care clinical photography using mobile devices is coming of age as a new standard of care for clinical documentation. High-quality cameras in modern smartphones facilitate faithful reproduction of clinical findings in photographs; however, clinical photographs captured on mobile devices are often taken using the native camera app on the device and transmitted using relatively insecure methods (eg, SMS text message and email) that do not preserve images as part of the electronic medical records. Native camera apps lack robust security features and direct integration with electronic health records (EHRs), which may limit patient acceptability and usefulness to clinicians. In March 2015, Mayo Clinic overcame these barriers by launching an internally developed mobile app that allows health care providers to securely capture clinical photographs and upload them to the EHR in a manner that is compliant with patient privacy and confidentiality regulations.ObjectiveThe study aimed to understand the perceptions, attitudes, and experiences of patients who were photographed using a mobile point-of-care clinical image capture app.MethodsThe study included a mail-out survey sent to 292 patients in Rochester, Minnesota, who were photographed using a mobile point-of-care clinical image capture app within a preceding 2-week period.ResultsThe surveys were completed by 71 patients who recalled being photographed. Patients were seen in 18 different departments, with the most common departments being dermatology (19/71, 27%), vascular medicine (17/71, 24%), and family medicine (10/71, 14%). Most patients (49/62, 79%) reported that photographs were taken to simply document the appearance of a clinical finding for future reference. Only 16% (10/62) of patients said the photographs were used to obtain advice from a specialist. Furthermore, 74% (51/69) of the patients said they would recommend medical photography to others and 67% (46/69) of them thought the photos favorably affected their care. Patients were largely indifferent about the device used for photography (mobile device vs professional camera; 40/69, 58%) or the identity of the photographer (provider vs professional photographer; 52/69, 75%). In addition, 90% (64/71) of patients found reuse of photographs for one-on-one learner education to be acceptable. Acceptability for other uses declined as the size of the audience increased, with only 42% (30/71) of patients deeming reuse on social media for medical education as appropriate. Only 3% (2/71) of patients expressed privacy or confidentiality concerns. Furthermore, 52% (33/63) of patients preferred to provide consent verbally, and 21% (13/63) of them did not think a specific consent process was necessary.ConclusionsPatient attitudes regarding medical photography using a secure EHR-integrated app were favorable. Patients perceived that photography improved their care despite the most common reason for photography being to simply document the appearance of a clinical finding for future reference. Whenever possible, health care providers should utilize secure EHR-integrated apps for point-of-care medical photography using mobile devices.

Method for automated generation of road accident scene sketch based on data from mobile device camera

The article proposes a method for the automated generation of a road traffic accident (RTA) scene sketch, which includes: video recording of vehicles involved in the traffic situation and the surroundings with a mobile device camera; map generation and reconstruction of the camera trajectory using the SLAM-method; comparison of the reconstructed 3D scene with the real dimensions of objects and their classification based on convolutional neural networks; setting the scale of the generated 3D scene and dimensioning the rest objects; conversion of the 3D model into a 2D RTA scene sketch.We present the results of a comparative analysis of the actual dimensions of objects at the RTA scene and their dimensions in the generated model, assessment of the accuracy of key objects’ classification, which allow us to state that the proposed solution may generate both a 3D RTA model and its 2D view in the form of an accident scene sketch. The obtained ratio between the actual dimensions of the surroundings and the 3D model makes it possible to calculate the distance between any objects and, if needed, to introduce data in the manual editing mode.