Variation In Classification Accuracy Research Articles

Comparative Performance Analysis of Metaheuristic Feature Selection Methods for Speech Emotion Recognition

Abstract Emotion recognition systems from speech signals are realized with the help of acoustic or spectral features. Acoustic analysis is the extraction of digital features from speech files using digital signal processing methods. Another method is the analysis of time-frequency images of speech using image processing. The size of the features obtained by acoustic analysis is in the thousands. Therefore, classification complexity increases and causes variation in classification accuracy. In feature selection, features unrelated to emotions are extracted from the feature space and are expected to contribute to the classifier performance. Traditional feature selection methods are mostly based on statistical analysis. Another feature selection method is the use of metaheuristic algorithms to detect and remove irrelevant features from the feature set. In this study, we compare the performance of metaheuristic feature selection algorithms for speech emotion recognition. For this purpose, a comparative analysis was performed on four different datasets, eight metaheuristics and three different classifiers. The results of the analysis show that the classification accuracy increases when the feature size is reduced. For all datasets, the highest accuracy was achieved with the support vector machine. The highest accuracy for the EMO-DB, EMOVA, eNTERFACE’05 and SAVEE datasets is 88.1%, 73.8%, 73.3% and 75.7%, respectively.

A Study on Dimensionality Reduction and Parameters for Hyperspectral Imagery Based on Manifold Learning.

With the rapid advancement of remote-sensing technology, the spectral information obtained from hyperspectral remote-sensing imagery has become increasingly rich, facilitating detailed spectral analysis of Earth's surface objects. However, the abundance of spectral information presents certain challenges for data processing, such as the "curse of dimensionality" leading to the "Hughes phenomenon", "strong correlation" due to high resolution, and "nonlinear characteristics" caused by varying surface reflectances. Consequently, dimensionality reduction of hyperspectral data emerges as a critical task. This paper begins by elucidating the principles and processes of hyperspectral image dimensionality reduction based on manifold theory and learning methods, in light of the nonlinear structures and features present in hyperspectral remote-sensing data, and formulates a dimensionality reduction process based on manifold learning. Subsequently, this study explores the capabilities of feature extraction and low-dimensional embedding for hyperspectral imagery using manifold learning approaches, including principal components analysis (PCA), multidimensional scaling (MDS), and linear discriminant analysis (LDA) for linear methods; and isometric mapping (Isomap), locally linear embedding (LLE), Laplacian eigenmaps (LE), Hessian locally linear embedding (HLLE), local tangent space alignment (LTSA), and maximum variance unfolding (MVU) for nonlinear methods, based on the Indian Pines hyperspectral dataset and Pavia University dataset. Furthermore, the paper investigates the optimal neighborhood computation time and overall algorithm runtime for feature extraction in hyperspectral imagery, varying by the choice of neighborhood k and intrinsic dimensionality d values across different manifold learning methods. Based on the outcomes of feature extraction, the study examines the classification experiments of various manifold learning methods, comparing and analyzing the variations in classification accuracy and Kappa coefficient with different selections of neighborhood k and intrinsic dimensionality d values. Building on this, the impact of selecting different bandwidths t for the Gaussian kernel in the LE method and different Lagrange multipliers λ for the MVU method on classification accuracy, given varying choices of neighborhood k and intrinsic dimensionality d, is explored. Through these experiments, the paper investigates the capability and effectiveness of different manifold learning methods in feature extraction and dimensionality reduction within hyperspectral imagery, as influenced by the selection of neighborhood k and intrinsic dimensionality d values, identifying the optimal neighborhood k and intrinsic dimensionality d value for each method. A comparison of classification accuracies reveals that the LTSA method yields superior classification results compared to other manifold learning approaches. The study demonstrates the advantages of manifold learning methods in processing hyperspectral image data, providing an experimental reference for subsequent research on hyperspectral image dimensionality reduction using manifold learning methods.

Remote Sensing Classification of Temperate Grassland in Eurasia Based on Normalized Difference Vegetation Index (NDVI) Time-Series Data

The Eurasian temperate grassland is the largest temperate grassland ecosystem and vegetation transition zone globally. The spatiotemporal distribution and changes of grassland types are vital for grassland monitoring and management. However, there is currently a lack of a unified classification method and standard distribution map of Eurasian temperate grassland types. The Normalized Difference Vegetation Index (NDVI) from remote sensing data is commonly used in grassland monitoring. In this paper, the Accumulated Rate of NDVI Change Index (ARNCI) was proposed to characterize the annual NDVI trend of different temperate grassland types, and four transitional categories were introduced to account for the overlap between them. Based on survey data on the distribution of Eurasian temperate grassland types in the 1980s, the study area was divided into three sub-regions: Northern China, Central Asia, and Mongolia. Regionally, pixel-based ARNCI maps in the 1980s and 1990s were successfully calculated from using NOAA’s AVHRR NDVI time-series products. The ARNCI classification thresholds for different sub-regions were determined, and classification experiments and validation were conducted for each sub-region. The overall accuracies of grasslands types classification for Northern China, Central Asia, and Mongolia in the 1980s were 75.3%, 64.2%, and 84.6%, respectively, which demonstrated that there were variations in classification accuracy in the three sub-regions, and the overall performance was favorable. Finally, distribution maps of Eurasian temperate grassland types in the 1980s and 1990s were obtained, and the spatiotemporal changes of grassland types were analyzed and discussed. The ARNCI method is simple to operate and easy to obtain data, and it can be conveniently used in grassland type classification. The maps firstly address the lack of remote sensing classification maps of Eurasian temperate grassland types, and provide a promising tool for monitoring grassland degradation, management, and utilization.

Supervised versus unsupervised approaches to classification of accelerometry data.

Sophisticated animal-borne sensor systems are increasingly providing novel insight into how animals behave and move. Despite their widespread use in ecology, the diversity and expanding quality and quantity of data they produce have created a need for robust analytical methods for biological interpretation. Machine learning tools are often used to meet this need. However, their relative effectiveness is not well known and, in the case of unsupervised tools, given that they do not use validation data, their accuracy can be difficult to assess. We evaluated the effectiveness of supervised (n = 6), semi-supervised (n = 1), and unsupervised (n = 2) approaches to analyzing accelerometry data collected from critically endangered California condors (Gymnogyps californianus). Unsupervised K-means and EM (expectation-maximization) clustering approaches performed poorly, with adequate classification accuracies of <0.8 but very low values for kappa statistics (range: -0.02 to 0.06). The semi-supervised nearest mean classifier was moderately effective at classification, with an overall classification accuracy of 0.61 but effective classification only of two of the four behavioral classes. Supervised random forest (RF) and k-nearest neighbor (kNN) machine learning models were most effective at classification across all behavior types, with overall accuracies >0.81. Kappa statistics were also highest for RF and kNN, in most cases substantially greater than for other modeling approaches. Unsupervised modeling, which is commonly used for the classification of a priori-defined behaviors in telemetry data, can provide useful information but likely is instead better suited to post hoc definition of generalized behavioral states. This work also shows the potential for substantial variation in classification accuracy among different machine learning approaches and among different metrics of accuracy. As such, when analyzing biotelemetry data, best practices appear to call for the evaluation of several machine learning techniques and several measures of accuracy for each dataset under consideration.

Multi-timepoint pattern analysis: Influence of personality and behavior on decoding context-dependent brain connectivity dynamics.

Behavioral traits are rarely considered in task‐evoked functional magnetic resonance imaging (MRI) studies, yet these traits can affect how an individual engages with the task, and thus lead to heterogeneity in task‐evoked brain responses. We aimed to investigate whether interindividual variation in behavior associates with the accuracy of predicting task‐evoked changes in the dynamics of functional brain connectivity measured with functional MRI. We developed a novel method called multi‐timepoint pattern analysis (MTPA), in which binary logistic regression classifiers were trained to distinguish rest from each of 7 tasks (i.e., social cognition, working memory, language, relational, motor, gambling, emotion) based on functional connectivity dynamics measured in 1,000 healthy adults. We found that connectivity dynamics for multiple pairs of large‐scale networks enabled individual classification between task and rest with accuracies exceeding 70%, with the most discriminatory connections relatively unique to each task. Crucially, interindividual variation in classification accuracy significantly associated with several behavioral, cognition and task performance measures. Classification between task and rest was generally more accurate for individuals with higher intelligence and task performance. Additionally, for some of the tasks, classification accuracy improved with lower perceived stress, lower aggression, higher alertness, and greater endurance. We conclude that heterogeneous dynamic adaptations of functional brain networks to changing cognitive demands can be reliably captured as linearly separable patterns by MTPA. Future studies should account for interindividual variation in behavior when investigating context‐dependent dynamic functional connectivity.

Identifying Sheep Activity from Tri-Axial Acceleration Signals Using a Moving Window Classification Model

Behaviour is a useful indicator of an individual animal’s overall wellbeing. There is widespread agreement that measuring and monitoring individual behaviour autonomously can provide valuable opportunities to trigger and refine on-farm management decisions. Conventionally, this has required visual observation of animals across a set time period. Technological advancements, such as animal-borne accelerometers, are offering 24/7 monitoring capability. Accelerometers have been used in research to quantify animal behaviours for a number of years. Now, technology and software developers, and more recently decision support platform providers, are integrating to offer commercial solutions for the extensive livestock industries. For these systems to function commercially, data must be captured, processed and analysed in sync with data acquisition. Practically, this requires a continuous stream of data or a duty cycled data segment and, from an analytics perspective, the application of moving window algorithms to derive the required classification. The aim of this study was to evaluate the application of a ‘clean state’ moving window behaviour state classification algorithm applied to 3, 5 and 10 second duration segments of data (including behaviour transitions), to categorise data emanating from collar, leg and ear mounted accelerometers on five Merino ewes. The model was successful at categorising grazing, standing, walking and lying behaviour classes with varying sensitivity, and no significant difference in model accuracy was observed between the three moving window lengths. The accuracy in identifying behaviour classes was highest for the ear-mounted sensor (86%–95%), followed by the collar-mounted sensor (67%–88%) and leg-mounted sensor (48%–94%). Between-sheep variations in classification accuracy confirm the sensor orientation is an important source of variation in all deployment modes. This research suggests a moving window classifier is capable of segregating continuous accelerometer signals into exclusive behaviour classes and may provide an appropriate data processing framework for commercial deployments.

Motor imagery-based neuro-feedback system using neuronal excitation of the active synapses

Neuronal excitation enables identifying the features of an electroencephalogram (EEG) signal for motor imagery detection. We propose a novel feature extraction algorithm supported by short-term cepstrum-based inverse filtration of neuronal excitation of the active synapse. The maximum power of the estimated neuronal excitation is subjected to a two-class Bayesian probabilistic classifier. The feature extraction algorithm with the Bayesian probabilistic classifier significantly improves the brain–computer interface performance compared with that of other conventional methods of EEG signal processing such as wavelet with a Bayesian classifier, autocorrelation and CSP filter with a naive Bayes classifier over the BCI competition II and IV datasets. Consequently, this neuronal excitation feature allows the authors to develop a motor imagery neuro-feedback system; the performance of which achieves 87.2% average classification accuracy, which is 14% greater than that of the wavelet-based algorithm and 6.2% greater than that of the TRSP-based algorithm, with 53 ms of processing time allotted for each instruction in a real-time experiment. However, brain signal variation across different subjects and sessions significantly impairs decision accuracy. Our neuronal excitation base feature extraction algorithm minimizes these variations in classification accuracy.

Evaluation of window size in classification of epileptic short-term EEG signals using a Brain Computer Interface software

The complexity of epilepsy created a fertile ground for further research in automated methods, attempting to help the epileptologists’ task. Over the past years, great breakthroughs have emerged in computer-aided analysis and the advent of Brain Computer Interface (BCI) systems has greatly facilitated the automated seizure analysis. In this study, an evaluation of the window size in automated seizure detection is proposed. The EEG signals from the University of Bonn was employed and segmented into 24 epochs of different window lengths with 50% overlap each time. Statistical and spectral features were extracted in the OpenViBE scenario that were used to train four different classifiers. Results in terms of accuracy were above 80% for the Decision Trees classifier. Also, results indicated that different window sizes provide small variations in classification accuracy.

Investigation on Land Cover Mapping Capability of Maximum Likelihood Classifier: A Case Study on North Canara, India

The success of a large number of real-world applications depends on the efficiency with which land use land cover classes are extracted from Remotely Sensed (RS) imagery. This study aims at analysing the performance of Maximum Likelihood Classifier (MLC) over multispectral remote sensor data characterized by the overlapping of spectral classes and presence of mixed pixels, investigating the impact of prior class probabilities on the overall classification performance, the impact of the number of land cover classes on classification accuracy, and identifying the decision parameters for assigning a pixel to a land cover class. A quantitative analysis of decision parameters of MLC for pixel assignment is presented for five randomly selected pixels in the study area. Landsat 8 multispectral data of North Canara District was collected from USGS website and is considered for the research. Seven land use land cover classes were identified over the study area. Results obtained indicate that use of prior probability estimates improves the classification accuracy. The study also presents the variation in classification accuracy for four different cases of total number of land cover classes. The conducted research is useful in understanding the performance of MLC under multiple scenarios.

Mapping per-pixel predicted accuracy of classified remote sensing images

The traditional approach of map accuracy assessment based on an error matrix does not capture the spatial variation in classification accuracy. Here, per-pixel accuracy prediction methods are proposed based on interpolating accuracy values from a test sample in which the reference class of each sampled pixel has been determined. Different accuracy prediction methods were developed based on four factors: predictive domain (spatial versus spectral), interpolation function (constant, linear, Gaussian, and logistic), incorporation of class information (interpolating each class separately versus grouping them together), and sample size. Developing accuracy maps using the spectral domain is a new approach in contrast to previous efforts based on the spatial domain. Performance of the prediction methods was evaluated using 26 test blocks, with 10km×10km dimensions, dispersed throughout the United States. Each block had complete coverage reference data manually extracted by interpreters and a land-cover map produced from Landsat imagery using a decision tree classification. The full scene maps were then compared to the corresponding reference maps to produce complete coverage accuracy information for each block. The predicted accuracy maps were produced from a sample of the reference data (i.e., the test dataset). The performance of the sample-based accuracy predictions was evaluated using the area under the curve (AUC) of the receiver operating characteristic. Relative to existing accuracy prediction methods, our proposed methods resulted in improvements of AUC of 0.15 or greater. Evaluation of the four factors comprising the accuracy prediction methods demonstrated that: i) interpolations should be done separately for each class instead of grouping all classes together; ii) if an all-classes approach is used, the spectral domain will result in substantially greater AUC than the spatial domain; iii) for the smaller sample size and per-class predictions, the spectral and spatial domain yielded similar AUC; iv) for the larger sample size (i.e., very dense spatial sample) and per-class predictions, the spatial domain yielded larger AUC; v) increasing the sample size improved accuracy predictions with a greater benefit accruing to the spatial domain; and vi) the function used for interpolation had the smallest effect on AUC. To conclude, the ability to produce per-pixel accuracy predictions yielding simple to understand accuracy maps opens up new possibilities for error propagation of remotely sensed products in a variety of disciplines.

Distinguishing Intensity Levels of Grassland Fertilization Using Vegetation Indices

Monitoring the reaction of grassland canopies on fertilizer application is of major importance to enable a well-adjusted management supporting a sustainable production of the grass crop. Up to date, grassland managers estimate the nutrient status and growth dynamics of grasslands by costly and time-consuming field surveys, which only provide low temporal and spatial data density. Grassland mapping using remotely-sensed Vegetation Indices (VIs) has the potential to contribute to solving these problems. In this study, we explored the potential of VIs for distinguishing five differently-fertilized grassland communities. Therefore, we collected spectral signatures of these communities in a long-term fertilization experiment (since 1941) in Germany throughout the growing seasons 2012–2014. Fifteen VIs were calculated and their seasonal developments investigated. Welch tests revealed that the accuracy of VIs for distinguishing these grassland communities varies throughout the growing season. Thus, the selection of the most promising single VI for grassland mapping was dependent on the date of the spectra acquisition. A random forests classification using all calculated VIs reduced variations in classification accuracy within the growing season and provided a higher overall precision of classification. Thus, we recommend a careful selection of VIs for grassland mapping or the utilization of temporally-stable methods, i.e., including a set of VIs in the random forests algorithm.

English

Knowledge mined from clinical data can be used for medical diagnosis and prognosis. By improving the quality of knowledge base, the efficiency of prediction of a knowledge-based system can be enhanced. Designing accurate and precise clinical decision support systems, which use the mined knowledge, is still a broad area of research. This work analyses the variation in classification accuracy for such knowledge-based systems using different rule lists. The purpose of this work is not to improve the prediction accuracy of a decision support system, but analyze the factors that influence the efficiency and design of the knowledge base in a rule-based decision support system. Three benchmark medical datasets are used. Rules are extracted using a supervised machine learning algorithm (PART). Each rule in the ruleset is validated using nine frequently used rule interestingness measures. After calculating the measure values, the rule lists are used for performance evaluation. Experimental results show variation in classification accuracy for different rule lists. Confidence and Laplace measures yield relatively superior accuracy: 81.188% for heart disease dataset and 78.255% for diabetes dataset. The accuracy of the knowledge-based prediction system is predominantly dependent on the organization of the ruleset. Rule length needs to be considered when deciding the rule ordering. Subset of a rule, or combination of rule elements, may form new rules and sometimes be a member of the rule list. Redundant rules should be eliminated. Prior knowledge about the domain will enable knowledge engineers to design a better knowledge base. ACM CCS (2012) Classification : Information systems→Information systems applications→Decision support systems→Expert systems *To cite this article: J. J. Christopher et al. , "Knowledge-based Systems and Interestingness Measures: Analysis with Clinical Datasets", CIT. Journal of Computing and Information Technology , vol. 24, no. 1, pp. 65-78, 2016.

Land Cover Classification Using ALOS Imagery For Penang, Malaysia

This paper presents the potential of integrating optical and radar remote sensing data to improve automatic land cover mapping. The analysis involved standard image processing, and consists of spectral signature extraction and application of a statistical decision rule to identify land cover categories. A maximum likelihood classifier is utilized to determine different land cover categories. Ground reference data from sites throughout the study area are collected for training and validation. The land cover information was extracted from the digital data using PCI Geomatica 10.3.2 software package. The variations in classification accuracy due to a number of radar imaging processing techniques are studied. The relationship between the processing window and the land classification is also investigated. The classification accuracies from the optical and radar feature combinations are studied. Our research finds that fusion of radar and optical significantly improved classification accuracies. This study indicates that the land cover/use can be mapped accurately by using this approach.

Forest tree species discrimination in western Himalaya using EO-1 Hyperion

The information acquired in the narrow bands of hyperspectral remote sensing data has potential to capture plant species spectral variability, thereby improving forest tree species mapping. This study assessed the utility of spaceborne EO-1 Hyperion data in discrimination and classification of broadleaved evergreen and conifer forest tree species in western Himalaya. The pre-processing of 242 bands of Hyperion data resulted into 160 noise-free and vertical stripe corrected reflectance bands. Of these, 29 bands were selected through step-wise exclusion of bands (Wilk's Lambda). Spectral Angle Mapper (SAM) and Support Vector Machine (SVM) algorithms were applied to the selected bands to assess their effectiveness in classification. SVM was also applied to broadband data (Landsat TM) to compare the variation in classification accuracy. All commonly occurring six gregarious tree species, viz., white oak, brown oak, chir pine, blue pine, cedar and fir in western Himalaya could be effectively discriminated. SVM produced a better species classification (overall accuracy 82.27%, kappa statistic 0.79) than SAM (overall accuracy 74.68%, kappa statistic 0.70). It was noticed that classification accuracy achieved with Hyperion bands was significantly higher than Landsat TM bands (overall accuracy 69.62%, kappa statistic 0.65). Study demonstrated the potential utility of narrow spectral bands of Hyperion data in discriminating tree species in a hilly terrain.

Classification Accuracy of Serum Apo A-I and S100B for the Diagnosis of Mild Traumatic Brain Injury and Prediction of Abnormal Initial Head Computed Tomography Scan

The objective of the current study was to determine the classification accuracy of serum S100B and apolipoprotein (apoA-I) for mild traumatic brain injury (mTBI) and abnormal initial head computed tomography (CT) scan, and to identify ethnic, racial, age, and sex variation in classification accuracy. We performed a prospective, multi-centered study of 787 patients with mTBI who presented to the emergency department within 6 h of injury and 467 controls who presented to the outpatient laboratory for routine blood work. Serum was analyzed for S100B and apoA-I. The outcomes were disease status (mTBI or control) and initial head CT scan. At cutoff values defined by 90% of controls, the specificity for mTBI using S100B (0.899 [95% confidence interval (CI): 0.78-0.92]) was similar to that using apoA-I (0.902 [0.87-0.93]), and the sensitivity using S100B (0.252 [0.22-0.28]) was similar to that using apoA-I (0.249 [0.22-0.28]). The area under the receiver operating characteristic curve (AUC) for the combination of S100B and apoA-I (0.738, 95% CI: 0.71, 0.77), however, was significantly higher than the AUC for S100B alone (0.709, 95% CI: 0.68, 0.74, p=0.001) and higher than the AUC for apoA-I alone (0.645, 95% CI: 0.61, 0.68, p<0.0001). The AUC for prediction of abnormal initial head CT scan using S100B was 0.694 (95%CI: 0.62, 0.77) and not significant for apoA-I. At a S100B cutoff of <0.060 μg/L, the sensitivity for abnormal head CT was 98%, and 22.9% of CT scans could have been avoided. There was significant age and race-related variation in the accuracy of S100B for the diagnosis of mTBI. The combined use of serum S100B and apoA-I maximizes classification accuracy for mTBI, but only S100B is needed to classify abnormal head CT scan. Because of significant subgroup variation in classification accuracy, age and race need to be considered when using S100B to classify subjects for mTBI.

Land-cover Classification Using Radarsat and Landsat Imagery for St. Louis, Missouri

This paper presents the potential of integrating radar data features with optical data to improve automatic land-cover mapping. For our study area of St. Louis, Missouri, Landsat ETMand Radarsat images are orthorectified and co-regis- tered to each other. A maximum likelihood classifier is utilized to determine different land-cover categories. Ground reference data from sites throughout the study area are collected for training and validation. The variations in classification accuracy due to a number of radar imaging processing techniques are studied. The relationship between the processing window and the land classification is also investigated. In addition, the Landsat images are fused with several combinations of processed radar features. The classification accuracies from the Landsat and radar feature combinations are studied. Our research finds that fusion of multi-sensor data improves the classification accuracy over a single Landsat sensor, although different processing techniques on radar images are required to obtain the best results. In our study, fusion of Landsat images and Radarsat feature combinations from a 13 � 13 entropy window, 9 � 9 data range widow, and 19 � 19 mean filter window achieves the highest overall accuracy improvement (10 percent) over the Landsat images alone.

Seasonal variation in classification accuracy of forest-cover types examined by a single band or band combinations

In vegetation remote sensing, classification accuracy cannot be fixed, due to seasonal variations in spectral reflectance characteristics. This study aims to clarify the seasonal variability of classification accuracy by forest-cover type. In particular, this paper describes seasonal variability by each band or band combinations. The study area is located in the vicinities of Hisayama and Sasaguri in Fukuoka Prefecture, Japan. Natural broadleaved, coniferous plantation, and bamboo forests were studied. Supervised classification was applied to six SPOT/HRV images taken in 1997. Kappa analysis was applied to assess the classification accuracy and compare any two error matrices. The results revealed that some single band or two-band combinations were as accurate as, or more accurate than, the full band (all three bands). The disadvantages of using a full band were especially apparent in the season with high classification accuracy. This study indicates that using all given bands does not necessarily result in the highest classification accuracy. This study also suggests that band selection within the scope of forest type and seasonal variability can contribute to better forest-cover-type classifications.

Using quadtree segmentation to support error modelling in categorical raster data

This paper explores the use of quadtree segmentation of a land-cover map to improve error modelling by (1) accounting for variation in classification accuracy among differently sized homogeneous map regions and (2) improving the statistical properties of map realizations generated by sequential indicator simulation (SIS). The latter was accomplished by locating the first simulation nodes—which affect many subsequent local simulations—within the largest quadtree leaves. These represent the largest homogeneous and hypothetically most accurately classified areas on the map. A case study showed that, indeed, the overall accuracy of the land-cover map increased with the quadtree leaf size, ranging from 67% for single cells in heterogeneous areas to 98% for homogeneous blocks of 256 cells. A map of the prior probability of each land-cover class was prepared on the basis of quadtree level-specific confusion matrices. Next, two unconditional SIS algorithms were used to generate sets of 50 realizations of the map, thereby accounting for spatial continuity of the residuals between indicator-transformed reference data and the priors. The proposed quadtree-guided SIS outperformed the more common multiple-steps method as judged by the reproduction of target proportions of map categories, class-specific accuracy levels and variogram reproduction.