Number Of Mappers Research Articles

E-Had: A distributed and collaborative detection framework for early detection of DDoS attacks

During the past few years, the traffic volume of legitimate traffic and attack traffic has increased manifolds up to Terabytes per second (Tbps). Because of the processing of such a huge traffic volume, it has become implausible to detect high rate attacks in time using conventional DDoS defense architectures. At present, the majority of the DDoS defense systems are deployed predominantly at the victim-end domain But these victim-end defense systems themselves are vulnerable to HR-DDoS attacks as the mammoth volume of attack traffic is generated by such type of attacks. The insufficient computational resources further make the problem more crucial at the victim-end. This paper proposed a distributed and collaborative architecture called E-Had that is capable of efficiently processing a large amount of data by distributing it among a number of mappers and reducers in a Hadoop based cluster. The proposed E-Had system has been comprehensively validated using various publicly available benchmarked datasets and real datasets generated in HA-DDoS testbed in terms of various detection system evaluation metrics. The experimental results clearly show that the proposed detection system is capable of early detection of different scenarios of DDoS attacks along with differentiating them from flash crowds.

Effects of Design Factors of HDFS on I/O Performance

Four major design factors of HDFS, the block size, the number of data nodes, the number of client processes and replication factor are investigated to find out the effects on the I/O performance of HDFS by performing experiments in a real physical HDFS infrastructure consisting of 64 Hadoop data nodes of Intel i9 based blades. The block size is observed to be optimal when it equals to about 1Gb or 128MB that is the amount of the data the hard disk drive device can effectively input and output for 1 second in most of today’s off-the-shelf computers. Sophisticated allocation strategy is required to determine the number of mappers and reducers as the number of data nodes increase because the overall performance is influenced in complicated manner by the number of raw data blocks of the job to be processed, the processing time of the blocks for each node and the overhead of shuffling. Experiments shows that Hadoop distributes the work properly that the number of clients does not have a significant impact as the number of clients increases. There is little delay in copying the replica because replication is done in pipelined manner although the network is overloaded.

P-Codec: Parallel Compressed File Decompression Algorithm for Hadoop

The increasing amount and size of data being handled by data analytic applications running on Hadoop has created a need for faster data processing. One of the effective methods for handling big data sizes is compression. Data compression not only makes network I/O processing faster, but also provides better utilization of resources. However, this approach defeats one of Hadoop’s main purposes, which is the parallelism of map and reduce tasks. The number of map tasks created is determined by the size of the file, so by compressing a large file, the number of mappers is reduced which in turn decreases parallelism. Consequently, standard Hadoop takes longer times to process. In this paper, we propose the design and implementation of a Parallel Compressed File Decompressor (P-Codec) that improves the performance of Hadoop when processing compressed data. P-Codec includes two modules; the first module decompresses data upon retrieval by a data node during the phase of uploading the data to the Hadoop Distributed File System (HDFS). This process reduces the runtime of a job by removing the burden of decompression during the MapReduce phase. The second P-Codec module is a decompressed map task divider that increases parallelism by dynamically changing the map task split sizes based on the size of the final decompressed block. Our experimental results using five different MapReduce benchmarks show an average improvement of approximately 80% compared to standard Hadoop.

Design and Experiment Analysis of a Hadoop-Based Video Transcoding System for Next-Generation Wireless Sensor Networks

The next-generation wireless sensor network (WSN) has the capability of carrying hundreds of high-definition video streams, beside the feature of massive employment of energy-efficient nodes. However, several challenges are identified with respect to the video bearing, such as the different video formats, enormous size of “raw” video, and compatibility with heterogeneous terminal devices. The video transcoding system (VTS) is widely believed to address these challenges. This paper introduces a cloud-based, more specifically, Hadoop-based, video transcoding system to fulfill the vision of bearing hundreds of HD video streams in the next generation WSN, with a discussion on optimization of several significant parameters. This paper obtains three remarkable results: (1) there is an optimal value of the number of Mappers; (2) the optimal value is closely related to the file size; (3) the transcoding time depends principally on the duration of video files rather than their sizes.