Built-in Camera Research Articles

A pilot study for a non-invasive system for detection of malignancy in canine subcutaneous and cutaneous masses using machine learning.

Early diagnosis of cancer enhances treatment planning and improves prognosis. Many masses presenting to veterinary clinics are difficult to diagnose without using invasive, time-consuming, and costly tests. Our objective was to perform a preliminary proof-of-concept for the HT Vista device, a novel artificial intelligence-based thermal imaging system, developed and designed to differentiate benign from malignant, cutaneous and subcutaneous masses in dogs. Forty-five dogs with a total of 69 masses were recruited. Each mass was clipped and heated by the HT Vista device. The heat emitted by the mass and its adjacent healthy tissue was automatically recorded using a built-in thermal camera. The thermal data from both areas were subsequently analyzed using an Artificial Intelligence algorithm. Cytology and/or biopsy results were later compared to the results obtained from the HT Vista system and used to train the algorithm. Validation was done using a "Leave One Out" cross-validation to determine the algorithm's performance. The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value of the system were 90%, 93%, 88%, 83%, and 95%, respectively for all masses. We propose that this novel system, with further development, could be used to provide a decision-support tool enabling clinicians to differentiate between benign lesions and those requiring additional diagnostics. Our study also provides a proof-of-concept for ongoing prospective trials for cancer diagnosis using advanced thermodynamics and machine learning procedures in companion dogs.

Tiny Machine Learning for High Accuracy Product Quality Inspection

The quality inspection of industrial products is a fundamental step in large-scale production as it boosts the yield and reduces the costs. Intelligent embedded platforms with built-in tiny machine learning (tinyML) algorithms and cameras can automate quality inspection; however, running complex deep learning algorithms in low-cost and low-power embedded devices is still challenging because of limited memory and energy resources. This article presents an innovative sensor system with three microcontroller unit (MCU)-based tinyML cameras capable of automatic artifact and anomaly detection in plastic components. The system consists of a top camera responsible for identifying shape defects and two side cameras for color anomalies. Data processing is executed locally with tinyML reducing data transmission to a few bytes. Two state-of-the-art convolutional neural network (CNN) architectures are evaluated, namely, MobileNetV2 and SqueezeNet. Results show how both the architectures—with appropriate compression techniques—are suitable to be evaluated by resource-constrained microcontrollers. The networks achieve 99% classification accuracy while maintaining suitable real-time performance, respectively, equal to 5 and 2 frames/s.

Calculation of Heartbeat Rate and SpO2 Parameters Using a Smartphone Camera: Analysis and Testing.

Mathematical and signal-processing methods were used to obtain reliable measurements of the heartbeat pulse rate and information on oxygen concentration in the blood using short video recordings of the index finger attached to a smartphone built-in camera. Various types of smartphones were used with different operating systems (e.g., iOS, Android) and capabilities. A range of processing algorithms were applied to the red-green-blue (RGB) component signals, including mean intensity calculation, moving average smoothing, and quadratic filtering based on the Savitzky-Golay filter. Two approaches-gradient and local maximum methods-were used to determine the pulse rate, which provided similar results. A fast Fourier transform was applied to the signal to correlate the signal's frequency components with the pulse rate. We resolved the signal into its DC and AC components to calculate the ratio-of-ratios of the AC and DC components of the red and green signals, a method which is often used to estimate the oxygen concentration in blood. A series of measurements were performed on healthy human subjects, producing reliable data that compared favorably to benchmark data obtained by commercial and medically approved oximeters. Furthermore, the effect of the video recording duration on the accuracy of the results was investigated.

Development of Automatic Portable Pathology Scanner and Its Evaluation for Clinical Practice.

Digital pathological scanners transform traditional glass slides into whole slide images (WSIs), which significantly improve the efficiency of pathological diagnosis and promote the development of digital pathology. However, the huge economic burden limits the spread and application of general WSI scanners in relatively remote and backward regions. In this paper, we develop an automatic portable cytopathology scanner based on mobile internet, Landing-Smart, to avert the above problems. Landing-Smart is a tiny device with a size of 208mm × 107mm × 104mm and a weight of 1.8kg, which integrates four main components including a smartphone, a glass slide carrier, an electric controller, and an optical imaging unit. By leveraging a simple optical imaging unit to substitute the sophisticated but complex conventional light microscope, the cost of Landing-Smart is less than $3000, much cheaper than general WSI scanners. On the one hand, Landing-Smart utilizes the built-in camera of the smartphone to acquire field of views (FoVs) in the section one by one. On the other hand, it uploads the images to the cloud server in real time via mobile internet, where the image processing and stitching method is implemented to generate the WSI of the cytological sample. The practical assessment of 209 cervical cytological specimens has demonstrated that Landing-Smart is comparable to general digital scanners in cytopathology diagnosis. Landing-Smart provides an effective tool for preliminary cytological screening in underdeveloped areas.

EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network

Measuring the level of engagement among participants in a meeting is crucial for evaluating collective understanding. While previous studies have utilized multiple sensors, such as wearable devices, to gauge engagement levels in offline environments, the shift to remote meetings due to the COVID-19 pandemic presents new challenges. In this study, we propose a method for measuring student engagement during online meetings using only the built-in web cameras on their devices. We collect high, middle, and low engagement level recording data from 24 students. We decided to collect data using the role-acting approach instead of conventional self-reporting or post-experiment annotation. With the feature extraction based approach, we achieved a classification rate of 46.7%. With a deep learning based approach, we achieved a classification rate of 89.5% using MobileNetV2 for leave-one-participant-out cross-validation, which demonstrated higher accuracy than previous studies. From the model, we implement an application <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">EnGauge</i> and conduct a pilot study. The results demonstrate a new approach to data collection, an optimal engagement level recognition model, and application scenarios.

Artificial Intelligence Systems for Supporting Video Surveillance Operators at International Airport

International airports come under critical infrastructure. Therefore, its security system consists of many crucial elements aimed at guaranteeing the safety of passengers and airport workers.There is a need to develop the safety aspects to the highest level. This paper aims to analyze the possibilities of supporting video surveillance operators at the airport with modern artificial intelligence (AI) solutions. To obtain the best research results, different functionalities were tested simultaneously. The first is based on built-in camera software to detect loitering, and the second is based on the already-used CCTV system. In addition, practical tests have been carried out at the International Katowice – Pyrzowice airport under actual conditions. The test results show that AI systems can help video surveillance operators perform their duties, but they must be refined.

Emotion Based Music Recommendation System

Abstract: We propose a new approach for playing music automatically using facial emotion. Most of the existing approaches involve playing music manually, using wearable computing devices, or classifying based on audio features. Instead, we propose to change the manual sorting and playing. We have used a Convolutional Neural Network for emotion detection. For music recommendations, Pygame &amp; Tkinter are used. Our proposed system tends to reduce the computational time involved in obtaining the results and the overall cost of the designed system, thereby increasing the system’s overall accuracy. Testing of the system is done on the FER2013 dataset. Facial expressions are captured using an inbuilt camera. Feature extraction is performed on input face images to detect emotions such as happy, angry, sad, surprise, and neutral. Automatically music playlist is generated by identifying the current emotion of the user. It yields better performance in terms of computational time, as compared to the algorithm in the existing literature

Comparison Analysis of Usability Using Controllers and Hand Tracking in Virtual Reality Gamelan (Sharon) Based On User Experience

Gamelan is one of the traditional musical instruments in Indonesia. Gamelan is also one of the cultures that must be preserved. To preserve gamelan, one way is by combining gamelan with technology that is currently developing. The technology that is currently trending is virtual reality. Virtual reality is often used for games in three-dimensional form. Gamelan can be something interesting in VR. In VR interactions, there are two interactions that are often used, namely controllers and hand tracking. Hand gestures are usually tracked using the built-in camera of the VR. Hand tracking can be used in VR applications to control the virtual object just like a controllers. This study aims to compare the use of controllers and hand tracking that involve interactions, namely collision, pressing, grabbing, and release. Thirty two Atma Jaya Yogyakarta University students who had tried virtual reality and played gamelan or had seen gamelan performances explore the differences between VR hand tracking and VR controllers in VR gamelan sharon. SUS (System Usability Scale) and USEQ (Use Questionnaire) were used to measure user usability and satisfaction. The result of the SUS score showed that users were more interested in using controllers than hand tracking. But the USEQ results showed that there were no difference in usefulness, ease of use, and satisfication. Based on the results of the interaction, using controllers looked more comfortable, but using hand tracking, playing gamelan looked more realistic.

Overview of Gesture Recognition Based Virtual Mouse and Keyboard

Abstract: Researchers around the globe nowadays are concentrating on making our devices more interactive and trying to make the devices more operational with minimal physical contact. Thai research proposes an interactive computer system that uses computer vision to create a virtual keyboard and mouse using only hand gestures. We can utilize a built-in camera or an external camera to track the image of different gestures performed by the person’s hand and according to different gestures conduct mouse cursor operations such as right and left clicks, as well as move the cursor. In addition to that, we can control the console using different gestures such as one finger to select the particular alphabet and a four-finger motion gesture to perform swipe operation in either the left or right direction. This proposed system with no wire or any other external device will act as virtual keyboard and mouse. The only hardware aspect of this system is the webcam which is used to capture images whereas the coding part will be done in python programming language

CAT S60 smartphone as a portable wound care device in home care

The purpose of this work is to study the suitability of the CAT S60 smartphone with built-in thermal camera to be used in self and home care to detect the risk level of wound appearance in advance. The purpose was to clarify different conditions where thermal imaging might act as a resource in detecting changes in limb circulation before visual signs even occur. The purpose is to detect early incipient tissue damage in foot usually occur in diabetic patients. Thermal images were acquired from voluntary domesticated elderly people. Thermal pictures from limbs of 3 persons were studied in order to find thermal differences indicating possible changes in limb circulation. Noteworthy thermal differences between limbs were found in elderly people. A smartphone having built-in thermal camera enables to detect plantar and limb thermal differences with a sufficient accuracy. This may support home monitoring for elderly people and thus reduce foot ulcers and possible foot amputations due to earlier detection and identification of harmful changes in limb circulation. Earlier detection of circulatory insufficiency via thermal imaging makes possible for nurses to intervene and enable medical assistance.

A novel motionless calibration method for augmented reality surgery navigation system based on optical tracker

Augmented reality (AR) surgery navigation systems display the pre-operation planned virtual model at the accurate position in the real surgical scene to assist the operation. Accurate calibration of the mapping relationship between the virtual coordinate and the real world is the key to the virtual-real fusion effect. Former calibration methods require the doctor user to conduct complex manual procedures before usage. This paper introduces a novel motionless virtual-real calibration method. The method only requires to take a mixed reality image containing both virtual and real marker balls using the built-in forward camera of the AR glasses. The mapping relationship between the virtual and real spaces is calculated by using the camera coordinate system as a transformation medium. The composition and working process of the AR navigation system is introduced, and then the mathematical principle of the calibration is designed. The feasibility of the proposed calibration scheme is verified with a verification experiment, and the average registration accuracy of the scheme is around 5.80mm, which is of same level of formerly reported methods. The proposed method is convenient and rapid to implement, and the calibration accuracy is not dependent on the user experience. Further, it can potentially realize the real-time update of the registration transformation matrix, which can improve the AR fusion accuracy when the AR glasses moves. This motionless calibration method has great potential to be applied in future clinical navigation research.

Combination of VSLAM and a Magnetic Fingerprint Map to Improve Accuracy of Indoor Positioning

With the continual advancement of positioning technology, people's use of mobile devices has increased substantially. The global navigation satellite system (GNSS) has improved outdoor positioning performance. However, it cannot effectively locate indoor users owing to signal masking effects. Common indoor positioning technologies include radio frequencies, image visions, and pedestrian dead reckoning. However, the advantages and disadvantages of each technology prevent a single indoor positioning technology from solving problems related to various environmental factors. In this study, a hybrid method was proposed to improve the accuracy of indoor positioning by combining visual simultaneous localization and mapping (VSLAM) with a magnetic fingerprint map. A smartphone was used as an experimental device, and a built-in camera and magnetic sensor were used to collect data on the characteristics of the indoor environment and to determine the effect of the magnetic field on the building structure. First, through the use of a preestablished indoor magnetic fingerprint map, the initial position was obtained using the weighted k-nearest neighbor matching method. Subsequently, combined with the VSLAM, the Oriented FAST and Rotated BRIEF (ORB) feature was used to calculate the indoor coordinates of a user. Finally, the optimal user's position was determined by employing loose coupling and coordinate constraints from a magnetic fingerprint map. The findings indicated that the indoor positioning accuracy could reach 0.5 to 0.7 m and that different brands and models of mobile devices could achieve the same accuracy.

Camera Image Processing on ESP32 Microcontroller with Help of Convolutional Neural Network

This paper analyzes a common ESP32 microcontroller with a built-in camera for image classification tasks using a convolutional neural network. ESP32 is commonly used in IoT devices to read data and control sensors, so its computing power is not significant, which has a positive effect on the cost of the device. The prevalence of ultra-low power embedded devices such as ESP32 will allow the widespread use of artificial intelligence built-in IoT devices. The duration of photographing and photo processing is obtained in the paper, as this can be a bottleneck of the microcontroller, especially together with machine learning algorithms. Deployed convolutional neural network, pre-trained on another device, MobileNet architecture on microcontroller and proved that ESP32 capacity is sufficient for simultaneous operation of both the camera and convolutional neural network.

A Metacognitive Technological Intervention for Promoting Eye Contact Among Children With ASD: Preliminary Research Evidence

This preliminary study aimed to examine improvement in eye contact among children with autism spectrum disorder (ASD) following an innovative technological intervention integrated with meta-cognitive guidance. Eighteen ASD participants, ages 5–9 years, were divided into two equal intervention groups—one received metacognitive guidance, and one did not. An eye-contact assessment was conducted pre-intervention and post-intervention by tracing the children’s head and eye movements using a built-in laptop camera. The intervention included six 30-min sessions in which the participants played a computer game (C-Me) that required them to make eye contact with cartoon characters who needed assistance in solving a problem or completing a task. The results indicate that the technological intervention effectively promoted eye contact among all participants, especially those in the metacognitive intervention group.

An origami microfluidic paper device for on-site assessment of urine tampering. First use of Nessler's reagent for the colorimetric determination of creatinine

The relevance of the problem of urine tampering is well-known in forensic toxicology, with sample dilution being the most used method to cheat toxicological controls. Among the criteria to assess urine integrity, the quantification of creatinine probably represents the most popular method. The present paper presents a simple and low-cost analytical device for on-site creatinine determination as first-line screening for urine dilution.The proposed microfluidic devices were designed as a three-dimensional origami pattern. The device included three colorimetric reactions based on picric acid (PA-based reagent), 3,5-dinitrobenzoic acid (DNBA-based reagent), and Nessler's reagent. The last one, to the best of our knowledge, has never been used before for creatinine determination. In order to assure the highest ease and economy of operation, the color detection and data processing were performed using a built-in smartphone camera and the associated software. The optimized device showed a detection limit of 0.02 g/L. The proposed method was used for the qualitative screening for urine dilution of 48 samples, showing a diagnostic sensitivity and specificity for PA-based, DNBA-based and Nessler's reagent of 83.3%–80.0%, 72.2%–70.0%, and 100.0%–93.3% respectively, versus reference enzymatic method adopting a cut-off of 0.2 g/L.In conclusion, the present preliminary study shows that the proposed device could be a useful tool for on-site screening for urine tampering at the time of sample collection for toxicological testing.

Facial muscle movements in patients with Parkinson's disease undergoing phonation tests.

Parkinson's disease (PD) is a serious neurodegenerative disease affecting the elderly. In general, the locomotion deficit, which seriously affects the daily life of patients with PD, usually occurs at a later stage. The mask face symptom meanwhile progressively worsens. However, facial muscle disorders and changes involved in the freezing mask are unclear. In this study, we recruited 35 patients with PD and 26 age- and sex-balanced controls to undergo phonation tests, while the built-in camera on the laptop recorded their facial expressions during the whole pronunciation process. Furthermore, FaceReader (version 7.0; Noldus Information Technology, Wageningen, Netherlands) was used to analyze changes in PD facial landmark movement and region movement. The two-tailed Student's t-test showed that the changes in facial landmark movement among 49 landmarks were significantly lower in patients with PD than in the control group (P < 0.05). The data on facial region movement revealed that the eyes and upper lip of patients with PD differed significantly from those in the control group. Patients with PD had defects in facial landmark movement and regional movement when producing a single syllable, double syllable, and multiple syllables, which may be related to reduced facial expressions in patients with PD.

Augmented Reality Based Interactive Cooking Guide.

Cooking at home is a critical survival skill. We propose a new cooking assistance system in which a user only needs to wear an all-in-one augmented reality (AR) headset without having to install any external sensors or devices in the kitchen. Utilizing the built-in camera and cutting-edge computer vision (CV) technology, the user can direct the AR headset to recognize available food ingredients by simply looking at them. Based on the types of the recognized food ingredients, suitable recipes are suggested accordingly. A step-by-step video tutorial providing details of the selected recipe is then displayed with the AR glasses. The user can conveniently interact with the proposed system using eight kinds of natural hand gestures without needing to touch any devices throughout the entire cooking process. Compared with the deep learning models ResNet and ResNeXt, experimental results show that the YOLOv5 achieves lower accuracy for ingredient recognition, but it can locate and classify multiple ingredients in one shot and make the scanning process easier for users. Twenty participants test the prototype system and provide feedback via two questionnaires. Based on the analysis results, 19 of the 20 participants would recommend others to use the proposed system, and all participants are overall satisfied with the prototype system.

МОДЕЛЮВАННЯ АВТОМАТИЧНОГО ПОЛЬОТУ МАЛОГО БЕЗПІЛОТНОГО ЛІТАЛЬНОГО АПАРАТУ НАД ЛІНІЄЮ-МАРКЕРОМ

The subject of study is the process of automating the behavior of an unmanned aerial vehicle during a flight following a given marker inside a room. The goal is to reduce the distance of deviation from the course and reduce the time to complete the flight along the trajectory set by the contrasting marker on the ground. The task: to conduct an analysis of existing libraries capable of performing automatic flight simulation and adaptation of the received code to the hardware platform of modern unmanned aerial vehicles; review the structure of the selected library, and modify the components of the library, which are necessary for the formation of automatic flight; create a model of the video processing and control system, check its operational parameters in a simulation environment. The methods used are: system analysis used to compare existing libraries of UAV flight simulation information systems within the framework of the assigned tasks, artificial intelligence methods for the development of a video stream data analysis subsystem and recognition of the provided marker during UAV flight simulation, system programming methods for automatic control of UAV flight based on the data obtained from the visual information analysis system, a method of simulation modeling to check the correctness of the developed algorithms in collaborative work. The following results were obtained. The choice of the library for the development and modeling of subsystems of automatic flight of a small unmanned aerial vehicle in a room is justified, a subsystem for processing video information obtained from the built-in camera of a UAV is created, an automatic flight control subsystem is developed taking into account the data of the video information analysis subsystem; a series of experiments was conducted on a simulation model to check the system's performance. Conclusions. In the course of the study, a model of the autonomous flight of a small unmanned aerial vehicle capable of flying over a contrasting color marker placed on the surface over which the UAV flies was improved. Unlike existing models, the system does not require the use of additional equipment, all computing operations must be performed on board the UAV, only the data of the video stream of the built-in camera and the coordinates of the UAV are transmitted to the control station, in addition, the minimum number of additional libraries necessary for creating a simulation model was used simulation modeling. This will reduce the time it takes to create a model, and, accordingly, conduct more experiments on improving parameters to achieve the goal of research.

Automatic Monitoring System for Seed Germination Test Based on Deep Learning

Germination test is an irreplaceable step in seed selection and breeding. The current traditional germination test method must rely on experienced professional technicians to repeatedly classify and count the germination status of seeds and count the germination rate at different moments during the whole test process (usually takes 2 to 10 days). Currently, only the German seed germination detection system (Germination Scanalyzer) can solve this problem, but it is so expensive that it has not been practically promoted. In order to improve breeding efficiency, an automatic monitoring system for seed germination tests based on deep learning was designed. It includes a modified germination thermostat, connected with a three-dimensional movable camera bin with built-in camera; a multifunctional software system capable of online, offline, and sentinel mode monitoring; a dense distributed small target detection algorithm (DDST-CenterNet) for seed germination monitoring systems. The system test results show that the seed germination test automatic monitoring system is low cost, does not depend on the seed background, light, camera model, and other usage environments, and has high scalability. The DDST-CenterNet algorithm proposed in this paper can still maintain high accuracy and good stability in the process of seed target detection and classification as the number and density of seeds increase, which is suitable for a special application scenario of seed germination test. In addition, the algorithm has high computational efficiency and can give detection results at a frame rate of not less than 10fps, which can be used in practical applications.

AI Virtual Mouse Using Hand Gesture Recognition

The mouse is one of HCI's (Human Computer Interaction) incredible inventions. A wireless mouse or a Bluetooth mouse still uses devices because a battery is needed for power and a dongle is needed to connect the mouse to the PC.Therefore, they are not totally device-free. The proposed AI virtual mouse system can solve this problem by using a webcam or built-in camera to record hand motions and identify hand tips using computer vision. The system's algorithm makes use of the machine learning algorithm. The computer can be controlled digitally and cando left-click, right-click, scrolling, and computer cursor functions based on hand motions without the need for a physical mouse. Deep learning is the basis for the hand detection method. As a result, by eliminating human contact and the requirement for external devices to operate the computer, the suggested strategy will stop the spread of COVID-19.