Median Absolute Deviation Method Research Articles

Construction of ellipsoid convex model of bounded uncertainties with outlier detection for application in non-probabilistic topology optimization

The ellipsoid convex model can be suitably used for structural analysis and design optimization under uncertain-but-bounded parameters and loads. Such a model can be constructed using measured samples of the uncertainties. However, the presence of outliers is often unavoidable due to system fluctuations in the measurements. Thus, it is necessary to detect any outliers among the samples before constructing the uncertainty model to prevent over-conservativeness. To this end, the present paper proposes a rational approach for constructing ellipsoid convex models with outlier detection. The concept of the local outlier factor (LOF) is utilized, in conjunction with the k-nearest natural neighbor to adaptively determine the neighborhood range. Then the outliers are detected using the scaled median absolute deviation method, based on the LOF values obtained. Finally, the minimum-volume ellipsoid convex model is constructed with a mathematically strict and efficient semi-definite programming formulation using the normal samples. The validity of the proposed approach is demonstrated with numerical examples. The application of the constructed uncertainty model in non-probabilistic robust topology optimization is demonstrated, and the results show the effectiveness of the proposed approach.

Risk assessment of ionizing radiation and radiological thresholds to compound an environmental baseline for the unconventional gas industry.

The exploration of unconventional hydrocarbons may be very effective in promoting economic development and confronting energy crisis around the world. However, the environmental risks associated with this practice might be an impediment if not adequately dimensioned. In this context, naturally occurring radioactive materials and ionizing radiation are sensitive aspects in the unconventional gas industry that may compromise the environmental sustainability of gas production and they should be properly monitored. This paper provides a radioecological assessment of the São Francisco Basin (Brazil) as part of an environmental baseline evaluation regarding the Brazilian potential for exploring its unconventional gas reserves. Eleven and thirteen samples of surface waters and groundwater were analyzed for gross alpha and beta using a gas flow proportional counter. A radiological background range was proposed using the ± 2 Median Absolute Deviation method. Using geoprocessing tools, the annual equivalent doses and lifetime cancer risk indexes were spatialized. Gross alpha and beta background thresholds in surface water ranged from 0.04-0.40Bq L-1 to 0.17-0.46Bq L-, respectively. Groundwater radiological background varies from 0.006-0.81Bq L-1 to 0.06-0.72Bq L-1 for gross alpha and beta, respectively. All environmental indexes are relatively higher in the south of the basin, probably a direct response to the local volcanic formations. Traçadal fault and local gas seepages might also influence the gross alpha and beta distribution. All samples have radiological indexes below the environmental thresholds, and should remain at acceptable levels with the development of the unconventional gas industry in Brazil.

Transformation Path of Modern Media from the Perspective of Internet of Things

The Internet of Things (IoT) is a new technology revolution that aims to link all ordinary physical items to the Internet, creating a massive worldwide network of uniquely connected things that can exchange information and fulfill planned activities, resulting in considerable advantages for consumers. Researchers have started to explore new techniques of technical help in various fields (e.g. health, transport, and education) as the Internet of Things has grown in popularity in recent years. In this research, we present a revolutionary IoT paradigm with an encryption mechanism to establish a kind of network that would allow the more intelligent media-data transfer. Initially, we collected COVID-19-related data and preprocessed the raw data using the median absolute deviation method. The preprocessed data that is to be transferred to the media is then stored in the IoT device using various sensors. To improve the security of the data, we propose Robust Modern Media Data Encryption (RMMDE) algorithm with Enhance Cuckoo Swarm Optimization (ECSO) algorithm. The suggested mechanism is compared to the traditional encryption approaches, and the metrics are evaluated using the OriginPro tool. This proposed methodology could be a start point for better and more efficient media data transmission.

Robust Calibration of MEMS Accelerometers in the Presence of Outliers

MEMS accelerometers suffer from different problems such as random and systematic errors and the presence of outliers in the measurements. Proper calibration is necessary to obtain accurate results. Various methods exist but very few are designed for robust calibration. This paper explores methods for calibrating MEMS accelerometers in the presence of outliers. Optimization techniques based on Levenberg–Marquardt algorithm and linearization are used to solve the nonlinear calibration equations. Outlier detection methods such as RANSAC (random sample consensus), the Mahalanobis distance, and the median absolute deviation (MAD) are integrated within the optimization algorithm. Outlier scores are calculated and used to eliminate outliers or assign weights. The experimental results show that outlier detection and removal methods allow to achieve substantial improvement in the calibration process compared with non-robust methods. The median absolute deviation is the most effective in detecting outliers. In the presence of outliers, the average error for MAD and model-based method is 0.0136 (a.u.). For non-robust methods, the error is 3.82 (a.u.). For the Mahalanobis distance, the error is 1.8778 (a.u.). The results show that the calibration error is the smallest for the model—based method. The Mahalanobis distance presents higher calibration errors and is less capable of detecting outliers; however, it is still better than non-robust methods. Finally Levenberg–Marquardt algorithm presents slightly better results than linearization.

Application of Object-Oriented Median Absolute Deviation Method to Building Extraction

Application of Object-Oriented Median Absolute Deviation Method to Building Extraction

MAD Control Chart for Autoregressive Models with Skew-Normal Distribution

AbstractThe major problem in analyzing control charts is to work with autocorrelated data. This problem can be solved by fitting a suitable model to the data and using the control chart for the residuals. The problem becomes very important, when the distribution of observation is nonnormal, in addition to being autocorrelated. Much recent research has focused on the development of appropriate statistical process control techniques for the autocorrelated data or nonnormal distribution, but few studies have considered monitoring the process mean of both nonnormal and autocorrelated data. In this paper, a simulation study is conducted to compare the performances of the control chart based on the median absolute deviation method (MAD) with those of existing control charts for the skew normal distribution. Simulation results indicate considerable improvement over existing control charts for nonnormal data can be achieved when the control charts with control limits based on the MAD method are used to monitor the process mean of nonnormal autocorrelated data.

The influence of the industrial area on the pollution outside its borders: a case study from Quintero and Puchuncavi districts, Chile.

The specific objectives of this study were: (1) to establish background content levels of potentially harmful elements in topsoil of Quintero and Puchuncavi districts; (2) to assess the degree of topsoil pollution and identify the local sources of pollution; and (3) to assess the health risk of metal exposure to population. Data from a Chilean Environmental Ministry report were analyzed. Multivariate analysis suggests that the potentially harmful elements are split into two main groups, which can be explained by controlling geogenic (Ni, V and Zn) and anthropogenic (As, Cd, Cu and Pb) factors. The spatial distribution of As, Cd, Cu and Pb displays a spatial pattern around the Puchuncavi-Ventanas industrial park. The background values determined with MAD (median absolute deviation) method include As 31.6mgkg-1, Cd 0.7mgkg-1, Cu 296mgkg-1, Ni 16.9mgkg-1, Pb 48.6mgkg-1, V 199mgkg-1 and Zn 192mgkg-1. The pollution indices reported considerable contamination to very high contamination in some soil samples. Hazard quotient values were higher for children than for adults, due to the ingestion and dermal absorption, while the cancer risk index values were in the acceptable range.

Anomaly delineation of porphyry copper deposits of Hanza Region through geochemical data analyses and multispectral remote sensing

Hanza region is located in the southern part of Urumieh–Dokhtar Metallogenic belt in southeastern Iran. This region includes six known porphyry copper deposits and it is considered as an ore- bearing region from geochemical point of view. The aim of this research is to examine effective processing techniques in the analysis of stream sediment geochemical datasets and ASTER satellite images. The processing methods have led to identification of eight new prospective areas. These methods are aimed at providing univariate geochemical maps. The stream sediment geochemical mapping for Cu and Mo was performed by the sample catchment basin approach. The results derived from this approach have been mapped in four classes associated with the first quartile, third quartile and threshold values obtained from Median Absolute Deviation method. False-colour composite and band ratio techniques were adopted as two well-known methods for the processing of an ASTER scene spanning the study area. Eight new targets for possible mineralization have been resulted from geochemical data analyses. Image processing techniques on the ASTER multispectral data have also revealed widespread hydrothermal alterations associated with the known porphyry copper deposits and the new prospects.

Energy management strategy in cloud computing: a perspective study

Optimizing energy consumption in cloud computing is yet a challenge despite the diversity of the proposed energy management strategies. Indeed, and during our related work study we have observed that the different elements or components which should be considered in order to be able to properly manage energy consumption in a cloud computing context are not well defined and/or discussed in terms of importance. This makes the proper classification and/or comparison of the different proposed strategies or techniques very difficult. Consequently, this paper aims, on the one hand, at defining and discussing properly such components in order to create a guideline and, on the other hand, to ease both the classification and the comparison of these proposed strategies and techniques. Second and after discussing some common weaknesses related to the current energy consumption optimization techniques and methods, this paper proposes energy-saving technique which uses a novel load detecting policy. This policy is based on the median absolute deviation method which uses the median and the standard deviation to calculate upper and lower thresholds which aim to classify hosts into either overloaded or under-loaded state. Simulation results have shown better results of the proposed technique compared to the existing ones especially in reducing energy consumption and the number of virtual machine migrations in addition to better active host time. Indeed, we found that the average of saved energy is around 40% compared to the built in techniques in cloudSim.

Erratum to: The Efficacy of Process Capability Indices Using Median Absolute Deviation and Their Bootstrap Confidence Intervals

The process capability indices (PCIs) $$\hbox {C}_{{p}}$$ and $$\hbox {C}_{{pk}}$$ are commonly used in industry to measure the process performance. The implementation of these indices required that process should follow a normal distribution. However, in many cases the underlying processes are non-normal which influence the performance of these indices. In this paper, median absolute deviation (MAD) is used as a robust measure of variability in two PCIs, $$\hbox {C}_{{p}}$$ and $$\hbox {C}_{{pk}}$$ . Extensive simulation experiments were performed to evaluate the performance of MAD-based PCIs under low, moderate and high asymmetric condition of Weibull, Log-Normal and Gamma distributions. The point estimation of MAD-based estimator of $$\hbox {C}_{{p}}$$ and $$\hbox {C}_{{pk}}$$ is encouraging and showed a good result in case of Log-Normal and Gamma distributions, whereas these estimators perform very well in case of Weibull distribution. The comparison of quantile method and MAD method showed that the performance of MAD-based PCIs is better for Weibull and Log-Normal processes under low and moderate asymmetric conditions, whereas its performance for Gamma distribution remained unsatisfactory. Four bootstrap confidence intervals (BCIs) such as standard (SB), percentile (PB), bias-corrected percentile (BCPB) and percentile-t (PTB) were constructed using quantile and MAD methods under all asymmetric conditions of three distributions under study. The bias-corrected percentile bootstrap confidence interval (BCPB) is recommended for a quantile (PC)-based PCIs, whereas CIs were recommended for MAD-based PCIs under all asymmetric conditions of Weibull, Log-Normal and Gamma distributions. A real-life example is also given to describe and validate the application of proposed methodology.

Detecting outliers in GARCH(p,q) models

ABSTRACTThis article is concerned with the outliers in GARCH models. An iterative procedure is given for testing the presence of any type of the four common outliers. Since the distribution of test statistic cannot be obtained analytically, its distributional behavior is investigated via a simulation study. The simulation study is based on estimation of residuals standard deviation (σν), which are obtained using two methods, median absolute deviation method (MAD), and omit-one method. The proposed procedure is employed for testing the presence of outliers in weekly light oil price Indexes of Iran during 1997 to 2010.

Exploration geochemistry data-application for anomaly separation based on discriminant function analysis in the Parkam porphyry system (Iran)

There are several statistical methodologies presented for separating anomalous values from background leading to determination of anomalous areas. These methods range from simple approaches to complicated ones. A common objective method for anomaly detection in geochemical exploration is target delineation by discriminant function analysis. Discriminant analysis (DA) is a multivariate statistical technique that classifies each observation into a specific group based on observed predictor variables and predefined groups. In the present study, to achieve this target, two populations of anomaly and background were determined using combination of Mahalanobis distance (MD) and median absolute deviation (MAD) method, using borehole data set. For this purpose, in the first step, Mahalanobis distance values should be determined and then MAD method should be applied on Mahalanobis distance values of boreholes data (grade of copper and molybdenum). Thus for separating anomaly from background of surface samples and determining anomaly areas, algorithm of DA method is applied on grade of Cu and Mo (surface samples). Results show that samples indicated by the DA methods as anomalous are more regular; less dispersed and are more accurate than other multivariate methods (e.g., MD method) and other univarite methods since anomalous samples are determined based on several variables. Finally, bivariate lithogeochemical map of the study area is provided for copper and molybdenum which has been prepared using DA method. In this map, the delineated Cu-Mo mineralization is closely associated with the defined zone of potassic alteration, which is also consistent with the field and microscopic observation of the Cu mineralization in this alteration zone. Moreover, it is associated with the phyllic alteration and is spatially conformable with the zone defined for it.

Comparing U-statistic and nonstructural methods for separating anomaly and generating geochemical anomaly maps of Cu and Mo in Parkam district, Kerman, Iran

In applied geochemistry, obtaining quantitative descriptions of geochemical patterns and identifying geochemical anomalies are important. To identify and separate geochemical anomalies, several statistical methodologies (nonstructural and structural) are presented by researchers. In this study, four nonstructural methods including threshold assessment method based on median and standard deviation, median absolute deviation (MAD), P.N product and Sinclair’s method are selected first and then U-statistic is considered as a structural method to compare their performance. Subsequently, the best method is used to assess prospective areas of Parkam district. Results show that P.N and Sinclair’s methods are not always efficient. MAD method reduced the background well and roughly increased the correlation factor of points. However, U-statistic method includes both mentioned advantages meaning in addition to reducing outlier data effect, it regularizes anomalous values and also their dispersion is reduced significantly. It is possible to determine anomaly areas according to anomalous samples positioning so that denser areas are more important. Finally, lithogeochemical map of study area is generated for copper and molybdenum. In this map, the Cu mineralization which is delineated by this method is closely associated with the defined potassic alteration zone (according to alteration map of the study area), and also, the delineated Mo mineralization is exactly associated with the phyllic alteration and is spatially conformable with the zone defined for it.

Assessment of prospective areas for providing the geochemical anomaly maps of lead and zinc in Parkam district, Kerman, Iran

There are several statistical methodologies presented for separating anomalous values from background leading to determination of anomalous areas. These methods range from simple approaches to complicated ones and include nonstructural and structural methods, subtraction separation method and so on. Structural methods take the sampling locations and their spatial relation into account for estimating the anomalous areas. The U-statistic method is one of the most important structural methods. It considers the location of samples and carries out the statistical analysis of the data without judging from a geochemical point of view and tries to separate subpopulations and also to determine anomalous areas. In the present study, several nonstructural methods including assessment of threshold based on median and standard deviation, median absolute deviation (MAD) and P.N product are used and U-statistic is considered as structural method to assess prospective areas of Parkam district. Results show that MAD method reduced background well and P.N method increased correlation of points. However, U-statistic method plays the role of both mentioned advantages meaning in addition to reducing outlier data effect, it regularizes anomalous values and also their dispersion is reduced significantly. It is possible to determine anomaly areas according to anomalous samples positioning so that denser areas are more important. Finally, lithogeochemical map of study area is provided for lead and zinc.

웨이블렛 변환 영역에서 MAD 순서통계량을 이용한 SAR 영상의 화질개선 구현

본 논문에서는 지형의 형태 파악에 주로 이용되는 SAR(Synthetic Aperture Radar) 영상의 화질을 저해하는 주된 요소인 잡음을 제거하기 위하여 웨이블렛 변환 기반 MAD순서통계량 알고리즘을 논의한다. 효과적인 영상개선을 위하여 SAR 영상에 근사부분대역의 웨이블렛 계수에 가중평균(Weighted average)법으로 영상처리하고 상세 부분대역의 웨이블렛 계수에 중앙절대편차(MAD : Median Absolute Deviation)를 이용한 임계값을 설정하여 왜곡요소를 제거하는 방법을 제안한다. 특히 제안 방법의 임계값은 잡음과 같은 왜곡요소를 배재하고 영상의 통계량을 고려하여 설정하였다. 제안된 방법은 실시간처리를 보장하기 위하여 DSP와 FPGA를 이용한 하드웨어로 구현하였으며 Xilinx FPGA를 사용하여 실험 하였다. This paper is proposed a wavelet-based the order statistics MAD(Median Absolute Deviation) method of SAR(Synthetic Aperture Radar) image for image enhancement. also The method of compared and defined the threshold the wavelet coefficients using MAD of the wavelet coefficients of the detail subbands was proposed to effectively image enhancement. In order to complement the disadvantage, the threshold of the proposed method sets up the image statistic and excludes the distortion. The hardware design is used FPGA of Xilinx and DSP system for the image enhancement and compressed encoding of the proposed algorithm. Therefore the proposed method is totally verified by comparing with the several other images.

MAD 순서통계량을 이용한 웨이블렛 변환기반 다중영상의 영상융합 및 JPEG2000 보드 구현

This paper is proposed a wavelet-based the order statistics MAD(Median Absolute Deviation) method of image fusion of Multi-image contaminated with visible image and infrared image. also The method of compared and defined the threshold the wavelet coefficients using MAD of the wavelet coefficients of the detail subbands was proposed to effectively fusion which of selected the high quality image of the two images. The existed fusion rule may be possible to get the distorted fusion image especially by the distortion in the relation between the pixel and indicator of two images in the existed fusion rules. In order to complement the disadvantage, the threshold of the proposed method sets up the image statistic and excludes the distortion. The hardware design is used FPGA of Xilinx and DSP system for the image fusion and compressed encoding of the proposed algorithm. Therefore the proposed method is totally verified by comparing with the several other multi-image and the proposed image fusion.

Estimation of the normal geomagnetic field and geological interpretation of the normal field anomalies over middle-northern Croatia

Based on the ground survey of total-field magnetic data at 53 sites, which recorded a time-series over the interval 2003–2005, the core field + long wavelength lithospheric field over the middle-northern Croatia region was calculated. The area of the survey was 18900 km2, an average distance between the neighbouring sites being 12 km. The results were reduced to 2004.5 epoch. This “normal” total-field was estimated using the firstorder Taylor polynomial as a function of geographic coordinates, and the polynomial coefficients were calculated with three methods of adjustment: simple and weighted least squares fits and adjustment according to the most frequent value. The stability of the normal field was tested using the Monte Carlo-type test, by decreasing the input data set in each adjustment (up to 90%). All obtained field residuals (measured-“normal”) were mutually compared, as well as fit coefficients of the Taylor polynomials. The residual values indicate the presence of shorter-wavelength anomalies, specifically three major residual anomalies were found (−106 nT, 74 nT and 57 nT). The geostatistical analysis of the ground survey data and the normal field residuals (respectively), using the median absolute deviation method, was further conducted in order to evaluate the calculated anomalies. The geological situation around the anomalies derived by the median absolute deviation method, and around the normal field residual anomalies, is given. The correlation was found between higher value anomalies of the normal field residuals, and shallow volcanic rocks and oil field, respectively.

Mapping of anomalies in continuous and discrete fields of stream sediment geochemical landscapes

ABSTRACT In this study, continuous field models of geochemical landscapes were obtained by interpolating stream sediment geochemical data while discrete field models of geochemical landscapes were obtained by attributing stream sediment geochemical data to their sample catchment basins. This study aimed to: (1) compare and contrast anomaly maps derived from continuous and discrete field models of stream sediment geochemical landscapes; and (2) determine which empirical frequency distributions – those of original point data or those of pixels values in models of stream sediment geochemical landscapes – are more useful in mapping of anomalies in such geochemical landscapes. Anomalies were mapped by using the mean+2 SDEV (standard deviation), median+2 MAD (median absolute deviation) and concentration–area (C–A) fractal methods of identifying threshold values in a geochemical data set. The results of the study in the Aroroy gold district (Philippines) highlight the following findings. In mapping of anomalies in either continuous or discrete field models of stream sediment geochemical landscapes, the C–A fractal method performs best, followed by the median+2 MAD method and then by the mean+2 SDEV method. Anomalies mapped in discrete field models, compared to anomalies mapped in continuous field models, of stream sediment geochemical landscapes mostly have stronger positive spatial associations with the known epithermal Au deposit occurrences in the study area. Empirical frequency distributions of either the original point data or the pixels values in the models of stream sediment geochemical landscapes are similarly useful in applying the C–A fractal method, but not in applying either the median+2 MAD or mean+2 SDEV method, to map anomalies in either continuous or discrete field models of such geochemical landscapes.

An improved algorithm for cleaning Ultra High-Frequency data

We develop a multiple-stage algorithm for detecting outliers in Ultra High-Frequency financial market data. We show that an efficient data filter needs to address four effects: the minimum tick size, the price level, the volatility of prices and the distribution of returns. We argue that previous studies tend to address only the distribution of returns, and may tend to ‘overscrub’ a data set. In this study, we address these issues in the market microstructure element of the algorithm. In the statistical element, we implement the robust median absolute deviation method to take into account the statistical properties of financial time series. The data filter is then tested against previous data-cleaning techniques and validated using a rich individual equity options transactions data set from the London International Financial Futures and Options Exchange. The paper has many relevant insights for any practitioner who uses high frequency derivatives data, for example, for market analysis or for developing trading strategies.

Lithological and pedological influences on the magnetic susceptibility of soil: Their consideration in magnetic pollution mapping

Magnetic susceptibility measurements are widely used to map and monitor the heavy metal pollution of soils. However, the magnetic properties of soils are influenced significantly by the bedrock lithology and soil-forming processes. Therefore, a main challenge in the data interpretation is to filter out the anthropogenic pollution signal. In this study we address this problem by analysing susceptibility values, heavy metal concentrations, as well as pedological parameters in a large soil data set from the eastern segment of Austria, covering a wide range of different lithologies and soil types. The statistic assessment demonstrates an influence of lithology and soil type on the magnetic susceptibility signal. Therefore anomalies are defined in sub sets of different soil types separately. Three different methods were applied to detect susceptibility anomalies: the median absolute deviation method, the boxplot method, and the population modelling method. These methods evaluate topsoil data only and can therefore also be applied to field measurements of magnetic susceptibility. The results were compared to the conventional method of calculating the difference of topsoil and subsoil susceptibility. All three approaches identify the main anomalies in the study area and are successful in circumventing the problem of erroneous anomaly definition due to pedological processes. However, knowledge of the lithological background is still necessary for a meaningful interpretation and can only be substituted by a large amount of data. The tested methods lead to thresholds of different height and therefore act as filters of different strength for the definition of anomalies.