TensorFlow Lite Research Articles

IMPLEMENTASI TENSOR FLOW LITE PADA TEACHABLE UNTUK IDENTIFIKASI TANAMAN AGLONEMA BERBASIS ANDROID

Aglonema or sri fortune has various types with various shapes, patterns and colors. Various types and more and more due to the many crossing processes carried out by owners and lovers of aglonema plants. For ordinary people who do not have insight into aglonema, it will be difficult to distinguish aglonema plants because the shapes, patterns and colors have similarities. It takes a Teachable Machine system with a complex but more sophisticated method that is able to recognize plants with a higher level of accuracy. The machine learning process is carried out on a computer to identify image data into classification results in the form of predictions. Tensorflow lite is a machine learning library specially designed for object recognition. Therefore, researchers are encouraged to create an Android-based mobile application that is able to recognize aglonema plants quickly, easily and accurately.

Edge Computing for Vision-Based, Urban-Insects Traps in the Context of Smart Cities.

Our aim is to promote the widespread use of electronic insect traps that report captured pests to a human-controlled agency. This work reports on edge-computing as applied to camera-based insect traps. We present a low-cost device with high power autonomy and an adequate picture quality that reports an internal image of the trap to a server and counts the insects it contains based on quantized and embedded deep-learning models. The paper compares different aspects of performance of three different edge devices, namely ESP32, Raspberry Pi Model 4 (RPi), and Google Coral, running a deep learning framework (TensorFlow Lite). All edge devices were able to process images and report accuracy in counting exceeding 95%, but at different rates and power consumption. Our findings suggest that ESP32 appears to be the best choice in the context of this application according to our policy for low-cost devices.

Compressed Deep Learning to Classify Arrhythmia in an Embedded Wearable Device.

The importance of an embedded wearable device with automatic detection and alarming cannot be overstated, given that 15–30% of patients with atrial fibrillation are reported to be asymptomatic. These asymptomatic patients do not seek medical care, hence traditional diagnostic tools including Holter are not effective for the further prevention of associated stroke or heart failure. This is likely to be more so in the era of COVID-19, in which patients become more reluctant on hospitalization and checkups. However, little literature is available on this important topic. For this reason, this study developed efficient deep learning with model compression, which is designed to use ECG data and classify arrhythmia in an embedded wearable device. ECG-signal data came from Korea University Anam Hospital in Seoul, Korea, with 28,308 unique patients (15,412 normal and 12,896 arrhythmia). Resnets and Mobilenets with model compression (TensorFlow Lite) were applied and compared for the diagnosis of arrhythmia in an embedded wearable device. The weight size of the compressed model registered a remarkable decrease from 743 MB to 76 KB (1/10000), whereas its performance was almost the same as its original counterpart. Resnet and Mobilenet were similar in terms of accuracy, i.e., Resnet-50 Hz (97.3) vs. Mo-bilenet-50 Hz (97.2), Resnet-100 Hz (98.2) vs. Mobilenet-100 Hz (97.9). Here, 50 Hz/100 Hz denotes the down-sampling rate. However, Resnets took more flash memory and longer inference time than did Mobilenets. In conclusion, Mobilenet would be a more efficient model than Resnet to classify arrhythmia in an embedded wearable device.

Open source deep learning inference libraries for autonomous driving systems

Automated driving functions, like highway driving and parking assist, are increasingly getting deployed in high-end cars with the goal of realizing self-driving car using Deep learning techniques like convolution neural network (CNN), Transformers. Camera based perception, Driver Monitoring, Driving Policy, Radar and Lidar perception are few of the examples built using DL algorithms in such systems. Traditionally custom software provided by silicon vendors are used to deploy these DL algorithms on devices. This custom software is very optimal for supported features (limited), but these are not flexible enough for evaluating various deep learning model architecture types quickly. In this paper we propose to use various open-source deep learning inference frameworks to quickly deploy any model architecture without any performance/latency impact. We have implemented this proposed solution with three open-source inference frameworks (Tensorflow Lite, TVM/Neo-AI-DLR and ONNX Runtime) on Linux running in ARM.

Low-Power On-Chip Implementation of Enhanced SVM Algorithm for Sensors Fusion-Based Activity Classification in Lightweighted Edge Devices

Smart homes assist users by providing convenient services from activity classification with the help of machine learning (ML) technology. However, most of the conventional high-performance ML algorithms require relatively high power consumption and memory usage due to their complex structure. Moreover, previous studies on lightweight ML/DL models for human activity classification still require relatively high resources for extremely resource-limited embedded systems; thus, they are inapplicable for smart homes’ embedded system environments. Therefore, in this study, we propose a low-power, memory-efficient, high-speed ML algorithm for smart home activity data classification suitable for an extremely resource-constrained environment. We propose a method for comprehending smart home activity data as image data, hence using the MNIST dataset as a substitute for real-world activity data. The proposed ML algorithm consists of three parts: data preprocessing, training, and classification. In data preprocessing, training data of the same label are grouped into further detailed clusters. The training process generates hyperplanes by accumulating and thresholding from each cluster of preprocessed data. Finally, the classification process classifies input data by calculating the similarity between the input data and each hyperplane using the bitwise-operation-based error function. We verified our algorithm on ‘Raspberry Pi 3’ and ‘STM32 Discovery board’ embedded systems by loading trained hyperplanes and performing classification on 1000 training data. Compared to a linear support vector machine implemented from Tensorflow Lite, the proposed algorithm improved memory usage to 15.41%, power consumption to 41.7%, performance up to 50.4%, and power per accuracy to 39.2%. Moreover, compared to a convolutional neural network model, the proposed model improved memory usage to 15.41%, power consumption to 61.17%, performance to 57.6%, and power per accuracy to 55.4%.

Human Monkeypox Classification from Skin Lesion Images with Deep Pre-trained Network using Mobile Application.

Recently, human monkeypox outbreaks have been reported in many countries. According to the reports and studies, quick determination and isolation of infected people are essential to reduce the spread rate. This study presents an Android mobile application that uses deep learning to assist this situation. The application has been developed with Android Studio using Java programming language and Android SDK 12. Video images gathered through the mobile device’s camera are dispatched to a deep convolutional neural network that runs on the same device. Camera2 API of the Android platform has been used for camera access and operations. The network then classifies images as positive or negative for monkeypox detection. The network’s training has been carried out using skin lesion images of monkeypox-infected people and other skin lesion images. For this purpose, a publicly available dataset and a deep transfer learning approach have been used. All training and testing steps have been applied on Matlab using different pre-trained networks. Then, the network that has the best accuracy has been recreated and trained using TensorFlow. The TensorFlow model has been adapted to mobile devices by converting to the TensorFlow Lite model. The TensorFlow Lite model has been then embedded into the mobile application together with the TensorFlow Lite library for monkeypox detection. The application has been run on three devices successfully. During the run-time, the inference times have been gathered. 197 ms, 91 ms, and 138 ms average inference times have been observed. The presented system allows people with body lesions to quickly make a preliminary diagnosis. Thus, monkeypox-infected people can be encouraged to act rapidly to see an expert for a definitive diagnosis. According to the test results, the system can classify the images with 91.11% accuracy. In addition, the proposed mobile application can be trained for the preliminary diagnosis of other skin diseases.

Anatomy of Deep Learning Image Classification and Object Detection on Commercial Edge Devices: A Case Study on Face Mask Detection

Developing efficient on-the-edge Deep Learning (DL) applications is a challenging and non-trivial task, as first different DL models need to be explored with different trade-offs between accuracy and complexity, second, various optimization options, frameworks and libraries are available that need to be explored, third, a wide range of edge devices are available with different computation and memory constraints. As such, trade-offs arise among inference time, energy consumption, efficiency (throughput/watt) and value (throughput/dollar). To shed some light in this problem, a case study is delivered where seven Image Classification (IC) and six Object Detection (OD) State-of-The-Art (SOTA) DL models were used to detect face masks on the following commercial off-the-shelf edge devices: Raspberry PI 4, Intel Neural Compute Stick 2, Jetson Nano, Jetson Xavier NX, and i.MX 8M Plus. First, a full end-to-end video pipeline face mask wearing detection architecture is developed. Then, the thirteen DL models were optimized, evaluated and compared on the edge devices, in terms of accuracy and inference time. To leverage the computational power of the edge devices, the models have been optimized, first, by using the SOTA optimization frameworks (TensorFlow Lite, OpenVINO, TensorRT, eIQ) and, second, by evaluating/comparing different optimization options, e.g., different levels of quantization. Note that the five edge devices are evaluated and compared too, in terms of inference time, value and efficiency. Last, we obtain insightful observations on which optimization frameworks, libraries and options to use and on how to select the right device depending on the target metric (inference time, efficiency and value). For example, we show that Jetson Xavier NX platform is the best in terms of latency and efficiency (FPS/Watt), while Jetson Nano is the best in terms of value (FPS/$).

Person Fall Detection System Based on Video Stream Analysis

The purpose of this paper is to develop an information system that would detect a fall based on video stream analysis. This work contains research and a description of the subject environment and the structure of the system. An analysis of current scientific works in this area was conducted, which allowed defining the purpose and primary objectives of the thesis. In addition, this analysis showed that this system would be in demand in the market of software that uses computer vision. The practical significance of the results can be determined by the fact that the proposed application performs on the level of other research works in this field and can be used by a broad market of customers. We applied pre-trained TensorFlow Lite CNN real-time model PoseNet to detect keypoints of a subject. Keypoints sequence is analyzed by LSTM as it works best when processing long sequences of data. The system was trained using a combination of the two biggest fall detection datasets. The system extracts the locations of 17 joint points of the human body and recognizes human movement by detecting the joint point location changes. By using joint points location as input to LSTM, the training and inference time was improved drastically. It also helped minimize the adverse effects of light sources and shadows

Custom Hardware Inference Accelerator for TensorFlow Lite for Microcontrollers

In recent years, the need for the efficient deployment of Neural Networks (NN) on edge devices has been steadily increasing. However, the high computational demand required for Machine Learning (ML) inference on tiny microcontroller-based IoT devices avoids a direct software deployment on such resource-constrained edge devices. Therefore, various custom and application-specific NN hardware accelerators have been proposed to enable real-time Machine Learning (ML) inference on low-power and resource-limited edge devices. Efficient mapping of the computational load onto hardware and software resources is a key challenge for performance improvement while keeping low power and a low area footprint. High performance and yet low power embedded processors may be attained via the usage of hardware acceleration. This paper presents an efficient hardware-software framework to accelerate machine learning inference on edge devices using a modified TensorFlow Lite for Microcontroller (TFLM) model running on a Microcontroller (MCU) and a dedicated Neural Processing Unit (NPU) custom hardware accelerator, referred to as MCU-NPU. The proposed framework supports weight compression of pruned quantized NN models and exploits the pruned model sparsity to reduce computational complexity further. The proposed methodology has been evaluated by employing the MCU-NPU acceleration for various TFLM-based NN architectures using the common MLPerf Tiny benchmark. Experimental results demonstrate a significant speedup of up to 724x compared to a pure software implementation. For example, the resulting runtime for the CIFAR-10 classification is reduced from about 20 sec to only 37 ms using the proposed hardware acceleration. Moreover, the proposed hardware accelerator outperforms all the reference models optimized for edge devices in terms of inference runtime.

Aplikasi Mobile Deteksi Dini Kanker Kulit Berdasarkan Image Processing

Currently, between 2 and 3 million non-melanoma skin cancers and 132,000 melanoma skin cancers occur globally each year (WHO, 2017). Skin cancer is one type of cancer that can cause death for many people. Because of this, an application is needed to easily detect skin cancer early that the cancer can be handled with more quickly. Besides, consultations with dermatologists have better prognosis (Avilés-Izquierdo et. al., 2016). Due to that, we built an early skin cancer detection application with dermatologist consultation. Our application helps to diagnose skin cancer before it grows into a life-threatening condition and is crucial to preserving lifestyle, future health, and aesthetics. Besides, thanks to online doctor consultations we have, however, getting diagnosed, prescribed and treated for your issues without spending time travelling to and from the doctors and waiting in queues can be just as effective. We used three management techniques such as machine learning to create data pipelines, build a model, and convert the model to TensorFlow lite with post-training quantization. Android to deploy the TensorFlow lite model and create the application. The application has a real-time connection using firebase. Moreover, cloud to create a simple database for doctor and diagnosis services on firebase.

이동감시 플랫폼을 위한 픽셀 및 GPS 데이터 기반 경량의 실시간 객체추적 알고리즘

This study proposed a method for detecting moving objects (e.g., people and vehicles) in the images of a lightweight mobile platform, and for the real-time visualization and tracking of the location information on the screen. First, a lightweight cable-driven platform and remote access environment were configured for the experiment. Next, a TensorFlow Lite environment was configured in the control computer, and an object detection algorithm was applied to the images obtained from the camera on the cable-driven platform. Subsequently, the bounding box coordinates of the object and class information were used as the inputs, as well as the camera specifications, GPS information, and the actual size information of the object to estimate the coordinates of the object. However, the significant limitations in the performance of CPU, GPU, and memory restrict the applications of high-performance deep learning models in a mobile platform environment. Therefore, in this study, a lightweight model, such as MobileNet SSD v2 was applied, and Google Edge TPU, which can use TensorFlow Lite environment, was used to facilitate computations, such as real-time object detection and coordinate estimation. In addition, this study presented a method for assigning a unique ID to a detected object and tracking it using the Euclidean distance, and the advantages of a simple algorithm using GPS data and a single moving camera in the surveillance field were presented.

Intelligence Eye for Blinds and Visually Impaired by Using Region-Based Convolutional Neural Network (R-CNN)

Intelligence Eye is an Android based mobile application developed to help blind and visually impaired users to detect light and objects. Intelligence Eye used Region-based Convolutional Neural Networks (R-CNN) to recognize objects in the object recognition module and a vibration feedback is provided according to the light value in the light detection module. A voice guidance is provided in the application to guide the users and announce the result of the object recognition. TensorFlow Lite is used to train the neural network model for object recognition in conjunction with extensible markup language (XML) and Java in Android Studio for the programming language. For future works, improvements can be made to enhance the functionality of the Intelligence Eye application by increasing the object detection capacity in the object recognition module, add menu settings for vibration intensity in light detection module and support multiple languages for the voice guidance.

Isi Piringku Dietary Meal Proportion Estimator Applications using SeeFood Image Segmentations

Isi Piringku atau My Meal Dish Content was a program initiated by the Indonesian ministry of health to promote a healthy daily lifestyle consisting of balanced dietary, enough hydration, active lifestyle, cleanliness and body weight control. The balanced diet meal was supposed to consist of ⅓ of carbohydrate intake, ⅓ of vegetable intake, ⅙ of fruit intake and ⅙ of protein intake every time. This introduces some difficulty that every meal must be measured to align with the dietary guidelines. This study targets estimating the meal diet proportion by its visual cues using smartphone application. While the actual meal content dietary division was weight based, for sake of simplicity the proportion in this study was estimated by each food area which roughly correlates to its volume. Using smartphone cameras in Android 9 Operating Systems and Tensorflow Lite Seefoods: Mobile Food Segmentation v1.0 module, an application was built to help users estimate their meal balances proportion. The original segmentation criterion was constructed using USDA dietary guidelines and it was reduced to only 4 food groups related to Isi Piringku criterion. Suggestion will be given regarding the segmentation result. The result was that the application was capable of estimating the meal diet proportion and giving suggestions based on the segmentation result. Although, the volume of the meal food groups estimated was still low on accuracy. This was correlated with the accuracy level of the segmentation module that was used. On average, the time needed to apply the segmentation process was around 2 to 3 seconds on a Snapdragon 835 device.

Traffic Light and Back-light Recognition using Deep Learning and Image Processing with Raspberry Pi

Traffic light detection and back-light recognition are essential research topics in the area of intelligent vehicles because they avoid vehicle collision and provide driver safety. Improved detection and semantic clarity may aid in the prevention of traffic accidents by self-driving cars at crowded junctions, thus improving overall driving safety. Complex traffic situations, on the other hand, make it more difficult for algorithms to identify and recognize objects. The latest state-of-the-art algorithms based on Deep Learning and Computer Vision are successfully addressing the majority of real-time problems for autonomous driving, such as detecting traffic signals, traffic signs, and pedestrians. We propose a combination of deep learning and image processing methods while using the MobileNetSSD (deep neural network architecture) model with transfer learning for real-time detection and identification of traffic lights and back-light. This inference model is obtained from frameworks such as Tensor-Flow and Tensor-Flow Lite which is trained on the COCO data. This study investigates the feasibility of executing object detection on the Raspberry Pi 3B+, a widely used embedded computing board. The algorithm’s performance is measured in terms of frames per second (FPS), accuracy, and inference time.

An embedded intelligence engine for driver drowsiness detection

Motor vehicle crashes involving drowsy driving are huge in number all over the world. Many studies revealed that 10%–30% of crashes are due to drowsy driving. Fatigue has costly effects on the safety, health, and quality of life. This drowsiness of drivers can be detected using various methods, for example, algorithms based on behavioural gestures, physiological signals and vitals. Also, few of them are vehicle based. Drowsiness of drivers was detected based on steering wheel movement and lane change patterns. A pattern is derived based on slow drifting and fast corrective steering movement. A prototype that detects the drowsiness of an automobile driver using artificial intelligence techniques, precisely using open-source tools like TensorFlow Lite on a Raspberry Pi development board, is developed. The TensorFlow model is trained on images captured from the video with the help of object detection using cascade classifier. In order to have a better accuracy, an Inception v3 architecture is used in pre-training the model with the image dataset. The final model is created and trained using long short-term memory and then the final TensorFlow model is converted to TensorFlow Lite model and this Lite model is used on Raspberry Pi board to detect the drowsiness of drivers. The results are comparable with desktop-based results in the literature.

Design and optimization of a TensorFlow Lite deep learning neural network for human activity recognition on a smartphone.

Human Activity Recognition (HAR), using machine learning to identify times spent (for example) walking, sitting, and standing, is widely used in health and wellness wearable devices, in ambient assistant living devices, and in rehabilitation. In this paper, a stacked Long Short-Term Memory (LSTM) structure is designed for HAR to be implemented on a smartphone. The use of an edge device for the processing means that the raw collected data does not need to be passed to the cloud for processing, mitigating potential bandwidth, power consumption, and privacy concerns. Our offline prototype model achieves 92.8% classification accuracy when classifying 6 activities using a public dataset. Quantization techniques are shown to reduce the model's weight representations to achieve a >30x model size reduction for improved use on a smartphone. The end result is an on-phone HAR model with accuracy of 92.7% and a memory footprint of 27 KB.

Pothole Detection and Reporting System

Abstract: Potholes on roads are the major problem for citizens acting as pedestrians as well as drivers. Government bodies which consist of engineers and workers are responsible to detect damages on roads and fix those damages. A recent study stated that every year around 3,597 people die due to potholes. The size and depth of the pothole may vary in a different place. Potholes had to be taken seriously. This system consists of a citizen with a handheld android/ios device with internet and GPS enabled, gathering the data in form of images and reporting to the government along with Geo-location. The study focuses on collecting and analyzing the datasets of potholes that are clicked by the users and detection of a pothole in that image via the TensorFlow Lite Model. The object detection system TensorFlow Lite is used for detecting the potholes, it shows that we can identify potholes from images clicked by the citizens and uploaded by the same application on the server and if a pothole is detected it ensures the complete reporting to the municipal authority along with the location. Index Term: Potholes, Android, GPS, Geo-location, Tensor- Flow Lite, Datasets, Reporting.

Real-Time classification of various types of falls and activities of daily livings based on CNN LSTM network

In this research, two multiclass models have been developed and implemented, namely, a standard long-short-term memory (LSTM) model and a Convolutional neural network (CNN) combined with LSTM (CNN-LSTM) model. Both models operate on raw acceleration data stored in the Sisfall public dataset. These models have been trained using the TensorFlow framework to classify and recognize among ten different events: five separate falls and five activities of daily livings (ADLs). An accuracy of more than 96% has been reached in the first 200 epochs of the training process. Furthermore, a real-time prototype for recognizing falls and ADLs has been implemented and developed using the TensorFlow lite framework and Raspberry PI, which resulted in an acceptable performance.

Smartphone-based real-time object recognition architecture for portable and constrained systems

Machine learning algorithms based on convolutional neural networks (CNNs) have recently been explored in a myriad of object detection applications. Nonetheless, many devices with limited computation resources and strict power consumption constraints are not suitable to run such algorithms designed for high-performance computers. Hence, a novel smartphone-based architecture intended for portable and constrained systems is designed and implemented to run CNN-based object recognition in real time and with high efficiency. The system is designed and optimised by leveraging the integration of the best of its kind from the state-of-the-art machine learning platforms including OpenCV, TensorFlow Lite, and Qualcomm Snapdragon informed by empirical testing and evaluation of each candidate framework in a comparable scenario with a high demanding neural network. The final system has been prototyped combining the strengths from these frameworks and led to a new machine learning-based object recognition execution environment embedded in a smartphone with advantageous performance compared with the previous frameworks.

Automated detection of COVID-19 from X-ray images using CNN and Android mobile

PurposeThe prevalence of the coronavirus disease 2019 (COVID-19) pandemic has made a huge impact on global health and the world economy. Easy detection of COVID-19 through any technological tool like a mobile phone can help a lot. In this research, we focus on detecting COVID-19 from X-ray images on Android mobile with the help of Artificial Intelligence (AI).MethodsA convolutional neural network (CNN) model is developed in MATLAB and then converted to the CNN model to TensorFlow Lite (TFLite) model to deploy on Android mobile. An Android application is developed which uses the TFLite model to detect COVID-19 using X-ray images.ResultsBy employing a 5-fold cross-validation, an average accuracy of 98.65%, sensitivity of 98.49%, specificity of 98.82%, precision of 98.81%, and F1-score of 98.65% are achieved in COVID-19 detection.ConclusionWith our developed Android application, users can detect COVID-19 from X-ray images on Android mobile, and it will be helpful for the diagnosis of COVID-19.