Small Batch Sizes Research Articles

New model for low-end computers: ResNet and VGG-16

ResNet-50 is a powerful architecture of convolutional neural networks, which gives truly high accuracy and very small error rate. However, this architecture seems not to be very effective when executing in low-end computers because of the small batch size for satisfying limited resources, which is not good for batch normalization. There is also an attempt to use VGG-16 as an alternative method, but vanishing gradients occur often. The proposed model is an improvement of VGG-16 using ResNet for shortcuts to prevent vanishing gradients, and the new architecture does not require batch normalization. As a result, the proposed model achieves a high test accuracy of 85.4%, while ResNet-50 achieves a test accuracy of 75.9% after 40 epochs of training 14,034 images from the Natural Scenes from Image Classification Challenge by Intel. This model is effective for applications related to image processing.

Heuristic Guided Hierarchical Reinforcement Learning Approach For The Economic Improvement Of Production Lines

The increasing variety of products poses a challenge for efficient manufacturing on production lines due to resulting small batch sizes and frequent product changes, which lower the average overall effectiveness. Especially for industries such as the Fast Moving Consumer Goods (FMCG) industry that manufacture at high speed on production lines, it is mandatory to increase the performance of production lines in an economical way. Due to the complexity of such production lines, identifying efficient actions and combining them into economic improvement trajectories is challenging. There are numerous approaches based on different concepts, such as simulation-based heuristics, to solve these challenges. In other application areas, reinforcement learning has shown remarkable success in recent years, and first reinforcement learning-based approaches to this specific problem can be found. However, these approaches mostly focus on details instead of providing a holistic view of the possibilities to improve a production line or are limited in their practical application due to lack of integration of existing expert knowledge or limited quality of results. For this reason, this paper proposes a hierarchical reinforcement learning approach that combines discrete event simulation with a heuristically driven multi-agent system. Thus, the selection of the improvement strategy of the production line is performed by one agent, and the dedicated improvement of specific parameters is performed by specialized subordinate other agents. Through this hierarchical multi-agent system, on the one hand, the learning rate can be increased. On the other hand, by guiding the agents through a heuristic based on expert knowledge, the learning quality is increased.

Empirical Evaluation of ECC Batch Verification Algorithms of Digital Signature in Wireless Sensor Network

Wireless sensor network (WSN) is an emerging topic in wireless communication where various sensor nodes are connected over the internet for data exchange. Exchanging the information over the WSN is very critical; therefore, it requires authentication and security mechanisms to ensure integrity and confidentiality. To avoid unauthorized use of WSN, the authentication through a digital signature turns into an essential task in WSN. Verification of individual signature is taking too much time and it is not suitable for wireless sensor network. Digital signature verification time can be reduced by using batch verification techniques in WSN. Therefore, the goal of this research is to examine and analyze the different type of batch verification algorithm based on Elliptic Curve Cryptography (ECC). With respect to security, the entire algorithm is identical to the standard Elliptic Curve Digital Signature Algorithm (ECDSA) verification algorithm. These algorithms are effectively limited to small batch sizes.

Efficient Transformer based Sentiment Classification Models

Recently, transformer models have gained significance as a state-of-the art technique for sentiment prediction based on text. Attention mechanism of transformer model speeds up the training process by allowing modelling of dependencies without regard to their distance in the input or output sequences. There are two types of transformer models – transformer base models and transformer large models. Since the implementation of large transformer models need better hardware and more training time, we propose new simpler models or weak learners with lower training time for sentiment classification in this work. These models enhance the speed of performance without compromising the classification accuracy. The proposed Efficient Transformer-based Sentiment Classification (ETSC) models are built by setting configuration of large models as minimum, shuffling dataset randomly and experimenting with various percentages of training data. Early stopping and smaller batch size in training techniques improve the accuracy of the proposed model. The proposed models exhibit promising performance in comparison with existing transformer-based sentiment classification models in terms of speed and accuracy.

Safety considerations for autonomous, modular robotics in aerospace manufacturing.

Industrial robots are versatile machines that can be used to implement numerous tasks. They have been successful in applications where-after integration and commissioning-a more or less static and repetitive behaviour in conjunction with closed work cells is sufficient. In aerospace manufacturing, robots still struggle to compete against either specialized machines or manual labour. This can be attributed to complex or custom parts and/or small batch sizes. Here, applicability of robots can be improved by enabling collaborative use-cases. When fixed protective fences are not desired due to handling problems of the large parts involved, sensor-based approaches like speed and separation monitoring (SSM) are required. This contribution is about how to construct dynamic volumes of space around a robot as well as around a person in the way that their combination satisfies required separation distance between robot and person. The goal was to minimize said distance by calculating volumes both adaptively and as precisely as possible given the available information. We used a voxel-based method to compute the robot safety space that includes worst-case breaking behaviour. We focused on providing a worst-case representation considering all possible breaking variations. Our approach to generate the person safety space is based on an outlook for 2D camera, AI-based workspace surveillance.

Power draw characteristics and comparison of scaling criteria for sawtooth impellers used in high shear mixing of shear thinning paint slurry

Power draw characteristics and scale-up procedures for high-speed sawtooth impellers were introduced by investigating various production sizes, fluids, impellers, and scale-up criteria to obtain the same product properties between different scales. The relationship between dimensionless Power number, Reynolds number, and Froude numbers were found for each impeller type at transitional flow regimes. The effect of Froude number was found to be very little, even though there was a deep vortex formation and no baffles in the mixing tank. Metzner – Otto constant which gives the relationship between rotational speed and shear rate was found as 20.7 and 12.4 for sawtooth impeller with big and small teeth, respectively. Productions were made by equalizing tip speed, shear rate, and power per volume between the commercial production and laboratory scales. In the result of rheological behavior measurement, stability test, and optical color evaluation, it was observed that the equal tip speed method gives a more intense dispersion in smaller batch sizes while the power per volume and shear rate method results in an insufficient particle size reduction performance.

No-code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context.

Industrial robots and cobots are widely deployed in most industrial sectors. However, robotic programming still needs a lot of time and effort in small batch sizes, and it demands specific expertise and special training, especially when various robotic platforms are required. Actual low-code or no-code robotic programming solutions are exorbitant and meager. This work proposes a novel approach for no-code robotic programming for end-users with adequate or no expertise in industrial robotic. The proposed method ensures intuitive and fast robotic programming by utilizing a finite state machine with three layers of natural interactions based on hand gesture, finger gesture, and voice recognition. The implemented system combines intelligent computer vision and voice control capabilities. Using a vision system, the human could transfer spatial information of a 3D point, lines, and trajectories using hand and finger gestures. The voice recognition system will assist the user in parametrizing robot parameters and interacting with the robot’s state machine. Furthermore, the proposed method will be validated and compared with state-of-the-art “Hand-Guiding” cobot devices within real-world experiments. The results obtained are auspicious, and indicate the capability of this novel approach for real-world deployment in an industrial context.

Equidistant Tool Path and Cartesian Trajectory Planning for Robotic Machining of Curved Freeform Surfaces

This article proposes an equidistant tool path planning strategy on curved freeform surfaces with the focus on robotic machining tasks. The strategy is based on the arc-length parameterization of the surface and can be considered as a variant of the iso-parametric path planning. This method allows us to directly consider a given Cartesian boundary in the parameter domain. Using an inverse interpolation scheme of the parameterization with cubic splines, the path planned in arc-length coordinates is mapped back in Cartesian coordinates. The equidistance property is also satisfied for larger path intervals and does not depend on the local curvature of the surface. The proposed trajectory planner ensures a constant velocity along the planned path on the freeform surface while limiting the maximum centripetal acceleration. However, general velocity profiles can be included depending on the actual machining application. Exploiting the differential geometric properties of the path and surface allows us to directly determine the orientation and angular velocity of the end-effector along the planned trajectory. The proposed methods are demonstrated with a robotic manipulator on a real curved freeform surface. Note to Practitioners—Robotic manipulators and five-axis computerized numerical control (CNC) machines provide high flexibility for machining processes on curved freeform surfaces, for example, polishing or grinding. For these applications, it is often desired that the distance between two subsequent machining paths is independent of the local geometry and surface curvature. This article proposes a novel method to plan equidistant coverage paths on a curved freeform surface. Therefore, we develop a transformation between the 3-D Cartesian coordinates and the 2-D parameters based on the arc-lengths along the surface. The geometry is given by CAD data or a 3-D scan of the part and is directly processed, making our approach particularly feasible for both large to small batch sizes and even individualized products. To cover an arbitrary area with parallel paths, its boundary—given in Cartesian coordinates—is transformed into arc-length parameters. The coverage path is planned in this parameter space, consisting of parallel, equidistant lines with interconnection paths. The path is transformed back in Cartesian coordinates using an inverse transformation. We plan the motion of the tool along this path, satisfying a desired feed rate depending on the actual machining application. The tool is slowed down when the maximum centripetal acceleration is reached. Considering the geometry of the path and the surface, we can also plan the tool’s orientation and its angular velocity during the manufacturing task. We demonstrate the path and motion planning approach on an experimental setup with a robotic manipulator.

Product-to-Product Virtual Metrology of Color Filter Processes in Panel Industry

The current thin film transistor liquid crystal display (TFT-LCD) panel industry evolves in transition from volume production to customized production service as a major competition strategy. Owing to the highly mixed production type of small batch sizes and high varieties, metrology data are usually collected too infrequently for building accurate virtual metrology (VM) models. In this paper, a new product-to-product (P2P) VM model to predict photoresist spacer heights in the color filter process of the array sector is developed. First, a position-based random forests model is proposed for the base product. Second, an ensemble model combining the base model and a biased estimation random forests model for the new product of small batch size is proposed to perform P2P VM between different products on the same tool. The real datasets of photoresist spacer heights for different products are used to illustrate the accuracy of the new P2P VM framework in the experimental study. Note to Practitioners—As global competition intensifies, production management needs to be adjusted to adapt swiftly to changes in the time-varying and competitive market. For manufacturing firms, how to operate the manufacturing system effectively and efficiently plays a crucial role in quality and yield improvements. The color filter process in the array sector is of critical importance in the TFT-LCD practice since it essentially dictates the quality level of subsequent processing steps in the cell and assembly sectors. Virtual metrology is a state-of-the-art measure that can be used for monitoring and controlling purposes to maintain the yield and productivity of the color filter process in the highly mixed production environment. An immediate challenge to be faced by the data scientists is how to build accurate virtual metrology models for a variety of product lines separately in color filter manufacturing. A new proposal of product-to-product virtual metrology is first addressed between products, enabling to build appropriate virtual metrology models for new products with small-sized production batches.

Adversarial momentum-contrastive pre-training

Recently proposed adversarial self-supervised learning methods usually require big batches and long training epochs to extract robust features, which will bring heavy computational overhead on platforms with limited resources. In order to help the network learn more powerful feature representations in smaller batches and fewer epochs, this paper proposes a novel adversarial momentum contrastive learning method, which introduces two memory banks corresponding to clean samples and adversarial samples, respectively. These memory banks can be dynamically incorporated into the training process to track invariant features among historical mini-batches. Compared with the previous adversarial pre-training model, our method achieves superior performance with smaller batch size and less training epochs. In addition, the model outperforms some state-of-the-art supervised defensive methods on multiple benchmark datasets after being fine-tuned on downstream classification tasks.

Innovative Online Measurement and Modelling Approach for Property-Controlled Flow Forming Processes

The production of complex multi-functional, high-strength parts is becoming increasingly important in the industry. Especially with small batch size, the incremental flow forming processes can be advantageous. The production of parts with complex geometry and locally graded material properties currently depicts a great challenge in the flow forming process. At this point, the usage of closed-loop control for the shape and properties could be a feasible new solution. The overall aim in this project is to establish an intelligent closed-loop control system for the wall thickness as well as the α’-martensite content of AISI 304L-workpieces in a flow forming process. To reach this goal, a novel sensor concept for online measurements of the wall thickness reduction and the martensite content during forming process is proposed. It includes the setup of a modified flow forming machine and the integration of the sensor system in the machine control. Additionally, a simulation model for the flow forming process is presented which describes the forming process with regard to the plastic workpiece deformation, the induced α’-martensite fraction, and the sensor behavior. This model was used for designing a closed-loop process control of the wall thickness reduction that was subsequently realized at the real plant including online measured feedback from the sensor system.

Ultraviolet Laser Sintering of Printed Nickel Oxide Nanoparticles for Thin-Film Thermistor via Aerosol Jet Printing Technology

In this study, nickel oxide (NiO) thin films were printed by an aerosol jet printer, which is capable of fabricating thin films on the curve substrate via air stream. To approach high efficiency fabricating thin film thermistors in small batch sizes, the printed NiO nanoparticle thin films were sintered by using a 355 nm wavelength ultraviolet (UV) laser; this novel fabrication method reduced several steps of the conventional manufacturing process of the thermistor. Compared with furnace heat treatments of the NiO thermistor in previous studies, the UV laser sintering not only significantly improved the electrical properties but decreased the treatment time from an hour to a second. Since the resistance declined, the thermistor has been operated at an ambient temperature, which provides ready measurement. The resistance and morphology of the thin films were analyzed for evaluating the effect of the laser treatment. To identify the proper UV laser parameters, three laser parameters, including laser output energy, frequency, and scanning speed, were studied. Due to the laser-sintering parameters, namely, 2 W, 150 mm/s, 90 kHz, and a B value of 4683 K, the resistance has been reduced from 106.8 MΩ to 6.15 MΩ at 100 °C. The experiments exhibited a series of analyses for sintering states and defects of printed NiO nanoparticle thin film, which were sintered by UV laser. For NiO nanoparticles, UV laser has higher absorption energy than that of other wavelength lasers, when excess laser output was applied to the NiO thin film, cracks were observed on the surface. It was found that the crystal plane distances were not affected by recrystallization, but the cracks were based on the XRD analysis. Based on the analysis, there were obvious regional compressive stains before the appearance of cracks, and the uneven shrinking strains caused the cracks on the surface as energy irradiation increased.

A precise nanoparticle quantification approach using microfluidics and single-particle tracking

Due to the limited available amounts of components, especially of low water-soluble drugs, formulation development is often impeded by a careful characterization. The use of small batch sizes might solve this problem but requires also adequate analytics. Concentration of nanoparticulate formulations lack straightforward evaluation techniques. In this work, a precise and straight-forward method is established to individually count nanoparticles. A microfluidic chip with known dimensions was used to visualize single particles flowing through the channel (single-particle tracking (SPT)). A sequence of 10,000 images was analyzed to determine the mean particle concentration. The proposed method is independent of the particular flow rate through the microfluidic chip as long as there is no particle overlap and a continuous exchange of particles. Monodisperse Rhodamine B labeled poly (methyl methacrylate) (PMMA) nanoparticles (267.03 ± 9.79 nm) were used as a model and reference particle system for the evaluation process of SPT allowing for a gravimetric determination based on density analysis using analytical ultracentrifugation (AUC) and gas pycnometry. The SPT method was evaluated and compared to other techniques used for concentration measurement. Both approaches (SPT and gravimetry) provide very similar and comparable results indicating the applicability of this novel quantification approach. In contrast, multi angle dynamic light scattering (MADLS) could not yield a precision as good as SPT (number density rel. standard deviation SD nSPT = 11.67%; SD nMADLS = 49.45%). Finally, the measured particle number concentrations can be realized in low concentration ranges (0.8249 μg mL−1 – 0.08249 μg mL−1) not accessible for MADLS (0.08249 mg mL−1 – 0.008249 mg mL−1) and gravimetric analysis.

Recursive SQL and GPU-support for in-database machine learning

In machine learning, continuously retraining a model guarantees accurate predictions based on the latest data as training input. But to retrieve the latest data from a database, time-consuming extraction is necessary as database systems have rarely been used for operations such as matrix algebra and gradient descent. In this work, we demonstrate that SQL with recursive tables makes it possible to express a complete machine learning pipeline out of data preprocessing, model training and its validation. To facilitate the specification of loss functions, we extend the code-generating database system Umbra by an operator for automatic differentiation for use within recursive tables: With the loss function expressed in SQL as a lambda function, Umbra generates machine code for each partial derivative. We further use automatic differentiation for a dedicated gradient descent operator, which generates LLVM code to train a user-specified model on GPUs. We fine-tune GPU kernels at hardware level to allow a higher throughput and propose non-blocking synchronisation of multiple units. In our evaluation, automatic differentiation accelerated the runtime by the number of cached subexpressions compared to compiling each derivative separately. Our GPU kernels with independent models allowed maximal throughput even for small batch sizes, making machine learning pipelines within SQL more competitive.

FARNN: FPGA-GPU Hybrid Acceleration Platform for Recurrent Neural Networks

GPU-based platforms provide high computation throughput for large mini-batch deep neural network computations. However, a large batch size may not be ideal for some situations, such as aiming at low latency, training on edge/mobile devices, partial retraining for personalization, and having irregular input sequence lengths. GPU performance suffers from low utilization especially for small-batch recurrent neural network (RNN) applications where sequential computations are required. In this article, we propose a hybrid architecture, called FARNN, which combines a GPU and an FPGA to accelerate RNN computation for small batch sizes. After separating RNN computations into GPU-efficient and GPU-inefficient tasks, we design special FPGA computation units that accelerate the GPU-inefficient RNN tasks. FARNN off-loads the GPU-inefficient tasks to the FPGA. We evaluate FARNN with synthetic RNN layers of various configurations on the Xilinx UltraScale+ FPGA and the NVIDIA P100 GPU in addition to evaluating it with real RNN applications. The evaluation result indicates that FARNN outperforms the P100 GPU platform for RNN training by up to 4.2 <inline-formula><tex-math notation="LaTeX">$\times {}$</tex-math></inline-formula> with small batch sizes, long input sequences, and many RNN cells per layer.

Exploiting Invariance in Training Deep Neural Networks

Inspired by two basic mechanisms in animal visual systems, we introduce a feature transform technique that imposes invariance properties in the training of deep neural networks. The resulting algorithm requires less parameter tuning, trains well with an initial learning rate 1.0, and easily generalizes to different tasks. We enforce scale invariance with local statistics in the data to align similar samples at diverse scales. To accelerate convergence, we enforce a GL(n)-invariance property with global statistics extracted from a batch such that the gradient descent solution should remain invariant under basis change. Profiling analysis shows our proposed modifications takes 5% of the computations of the underlying convolution layer. Tested on convolutional networks and transformer networks, our proposed technique requires fewer iterations to train, surpasses all baselines by a large margin, seamlessly works on both small and large batch size training, and applies to different computer vision and language tasks.

Evaluation of neural network models for landslide susceptibility assessment

ABSTRACT Identifying and assessing the disaster risk of landslide-prone regions is very critical for disaster prevention and mitigation. Owning to their special advantages, neural network algorithms have been widely used for landslide susceptibility mapping (LSM) in recent decades. In the present study, three advanced neural network models popularly used in relevant studies, i.e. artificial neural network (ANN), one dimensional convolutional neural network (1D CNN) and recurrent neural network (RNN), were evaluated and compared for LSM practice over the Qingchuan County, Sichuan province, China. Extensive experimental results demonstrated satisfactory performances of these three neural network models in accurately predicting susceptible regions. Specifically, ANN and 1D CNN models yielded quite consistent LSM results but slightly differed from those of RNN model spatially. Nevertheless, accuracy evaluations revealed that the RNN model outperformed the other two models both qualitatively and quantitatively but its complexity was relatively high. Experiments concerning training hyper-parameters on the performance of neural network models for LSM suggested that relatively small batch size values with Tanh activation function and SGD optimizer are essential to improve the performance of neural network models for LSM, which may provide a thread to help those who apply these advanced algorithms to improve their efficiency.

Instance Segmentation Based on Improved Self-Adaptive Normalization.

The single batch normalization (BN) method is commonly used in the instance segmentation algorithms. The batch size is concerned with some drawbacks. A too small sample batch size leads to a sharp drop in accuracy, but a too large batch may result in the memory overflow of graphic processing units (GPU). These problems make BN not feasible to some instance segmentation tasks with inappropriate batch sizes. The self-adaptive normalization (SN) method, with an adaptive weight loss layer, shows good performance in instance segmentation algorithms, such as the YOLACT. However, the parameter averaging mechanism in the SN method is prone to problems in the weight learning and assignment process. In response to such a problem, the paper proposes to replace the single BN with an adaptive weight loss layer in SN models, based on which a weight learning method is developed. The proposed method increases the input feature expression ability of the subsequent layers. By building a Pytorch deep learning framework, the proposed method is validated in the MS-COCO data set and Autonomous Driving Cityscapes data set. The experimental results prove that the proposed method is effective in processing samples independent from the batch size. The stable accuracy for all kinds of target segmentation is achieved, and the overall loss value is significantly reduced at the same time. The convergence speed of the network is also improved.

Automatic Detection and Localization of Pulmonary Nodules in CT Images Based on YOLOv5

&lt;p&gt;Lung cancer has always threatening human health and life. As small pulmonary nodules are main early features of lung cancer, early screening for small pulmonary nodules through computed tomography (CT) imaging is essential for the treatment of lung cancer. In this paper, the YOLOv5 model is improved to improve the ability of detection and recognition of small pulmonary nodules in complex CT lung images. Firstly, the preprocessing step is put into effect to obtain the lung parenchyma in CT images. Then, the backbone structure of YOLOv5 is improved by iResNet to improve the ability of feature extraction, and the feature fusion network is improved by BiFPN to improve the detection ability of small pulmonary nodules. Finally, the strategy of group normalization is used to improve the model performance under small batch size training condition. The experimental results on LUNA16 data set show that the detection AP of the improved model reach 94.8%, the competitive index score is 0.895, and the sensitivity is 78.1%, 94.4%, under 1/8 and 1/4 FPs, respectively. Compared with other two-dimensional target detection models, the improved yolov5 model has better detection ability of small pulmonary nodules. And, the results are better than most other two-dimensional pulmonary nodule detection methods. In addition, compared with other three-dimensional pulmonary nodule detection methods.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

Development of a Toolchain for Automated Optical 3D Metrology Tasks

Modern manufacturing processes are characterized by growing product diversities and complexities alike. As a result, the demand for fast and flexible process automation is ever increasing. However, higher individuality and smaller batch sizes hamper the use of standard robotic automation systems, which are well suited for repetitive tasks but struggle in unknown environments. Modern manipulators, such as collaborative industrial robots, provide extended capabilities for flexible automation. In this paper, an adaptive ROS-based end-to-end toolchain for vision-guided robotic process automation is presented. The processing steps comprise several consecutive tasks: CAD-based object registration, pose generation for sensor-guided applications, trajectory generation for the robotic manipulator, the execution of sensor-guided robotic processes, test and the evaluation of the results. The main benefits of the ROS framework are readily applicable tools for digital twin functionalities and established interfaces for various manipulator systems. To prove the validity of this approach, an application example for surface reconstruction was implemented with a 3D vision system. In this example, feature extraction is the basis for viewpoint generation, which, in turn, defines robotic trajectories to perform the inspection task. Two different feature point extraction algorithms using neural networks and Voronoi covariance measures, respectively, are implemented and evaluated to demonstrate the versatility of the proposed toolchain. The results showed that complex geometries can be automatically reconstructed, and they outperformed a standard method used as a reference. Hence, extensions to other vision-controlled applications seem to be feasible.