Amazon S3 Research Articles

AI/ML Drone-based Arecanut Monitoring System (ADAMS)

The Indian economy relies heavily on agriculture with 72% of farmers in the states of Karnataka and Kerala producing 330 million kilograms of arecanuts. However, a rapid increase in young adults leaving the agriculture field brings hardships for older farmers like my grandparents since physical labor is increasingly expensive, dangerous, and scarce. The purpose of this research is to automate arecanut crop monitoring by providing a safe, affordable, user-friendly, and cutting-edge system that utilizes six major elements to achieve the goal: arecanut farm, drone, dataset collection, data annotations, AI/ML Mask R-CNN prediction model, and app. Research about various drones were conducted to ensure the drone abides by Indian UAV regulations. Since there were no pre-existing arecanut datasets, a DJI Mini 2 drone and Canon DSLR Camera were utilized to capture over 2000 arecanut images in India. Creating my own dataset from scratch was challenging, however it was a significant contribution in advancing computer vision. This dataset was stored on Amazon S3 and classified into ripe, unripe, and dry arecanuts. Each image was manually annotated using VGG Image Annotator and passed into a Convolutional Neural Network called Mask R-CNN to create a prediction model. This was trained on Amazon Web Services (AWS) for 100 epochs. Overall, epoch 85 gave accurate predictions. Additionally, instead of the DJI Mini 2 drone, the Skydio drone would be the best option for real world implementations. My research is a practical and innovative solution to alleviate farmers’ financial and physical hardships.

Multimodal Remote Sensing and Machine Learning for Precision Agriculture: A Review

This study proposed various machine learning and deep learning techniques to integrate and analyze varieties of data in precision agriculture systems. Agricultural systems have undergone a digital transformation, which has resulted in the evolution of many management components into artificially intelligent systems to better value the ever-increasing amounts of data. In the process of putting in place farming systems that are based on knowledge, several obstacles can be overcome using machine learning. The data obtained are transmitted to on-site storage, where extraction, loading, and transformation are performed. The data is preprocessed and transferred to the AWS (Amazon Web Services) cloud (Amazon S3 Bucket). The best model is deployed such that new data can be fit into the model to make adequate prediction or classification. Such a solution can be adapted by building an algorithm to simulate the AWS machine learning technique. A small-scale pilot project can be executed, and the output of the prediction or classification model can be displayed using a web-based software or mobile app.

An Introduction to Cordra

Cordra is a digital object server that can function as a key infrastructural piece in FAIR DO (findable, accessible, interoperable and reusable digital object) implementations. Cordra manages JSON records and payloads as typed digital objects identified by handles. Cordra is neither a database nor an indexer, but it integrates the two and provides a unified interface. Cordra is intended to support both quick prototyping as well as production systems. For prototyping, Cordra makes it easy to get up and running rapidly with a digital object server. A potential Cordra administrator can download Cordra and very quickly have a server which supports creation, search, and retrieval of digital objects with resolvable identifiers. The server supports Digital Object Interface Protocol (DOIP) and HTTP APIs out of the box, as well as an immediately usable prototype user interface. Cordra saves substantial development time as it comes with ready-made functionality ranging from user authentication and access control to information validation, enrichment, storing, and indexing. By default, Cordra is configured to store objects on the local file system of the machine and use embedded Apache Lucene for indexing. Simply by editing type definitions in Cordra's user interface, the administrator can start changing the behavior of the APIs and user interface in real time for experimentation, including adding custom operations. For production use, Cordra allows intensive extension and customization of the processes underlying the digital object server: how digital objects are stored and indexed, how they are validated and enriched, how users authenticate, when and to whom to give access to objects, and what custom operations can be performed. In production Cordra is run at scale, supporting high reliability and performance; among other options Cordra supports MongoDB and Amazon S3 for storage, and Elasticsearch and Apache Solr for indexing. By definition of the underlying types and operations, Cordra is intended to serve directly as the API backend for a production application. This talk will cover basic Cordra features as well as customization/configuration basics. Examples of current use will be shown, including the use of the Digital Object Interface Protocol (DOIP), for which Cordra serves as a reference implementation. Current users of Cordra include the Derivatives Service Bureau (DSB), which uses Cordra as part of its backend to manage the automated generation of International Securities Identification Numbers (ISINs) for OTC derivatives in the financial services sector; and the British Standard Institute (BSI) whose Identify service for construction product manufacturers aims to assign a Universal Persistent Identification Number (UPIN) "for every product that is specified and incorporated in a building structure". The DSB, Cordra users since 2017, has a production system with over 80 million identified digital objects which receives millions of searches each month. BSI.Identify has a system where Cordra's DOIP interface is directly accessible as the service's public API.

Blockchain-Based Decentralized Cloud Solutions for Data Transfer.

Cloud computing has increased its service area and user experience above traditional platforms through virtualization and resource integration, resulting in substantial economic and societal advantages. Cloud computing is experiencing a significant security and trust dilemma, requiring a trust-enabled transaction environment. The typical cloud trust model is centralized, resulting in high maintenance costs, network congestion, and even single-point failure. Also, due to a lack of openness and traceability, trust rating findings are not universally acknowledged. “Blockchain is a novel, decentralised computing system. Its unique operational principles and record traceability assure the transaction data's integrity, undeniability, and security. So, blockchain is ideal for building a distributed and decentralised trust infrastructure. This study addresses the difficulty of transferring data and related permission policies from the cloud to the distributed file systems (DFS). Our aims include moving the data files from the cloud to the distributed file system and developing a cloud policy. This study addresses the difficulty of transferring data and related permission policies from the cloud to the DFS. In DFS, no node is given the privilege, and storage of all the data is dependent on content-addressing. The data files are moved from Amazon S3 buckets to the interplanetary file system (IPFS). In DFS, no node is given the privilege, and storage of all the data is dependent on content-addressing.

WukaStore: Scalable, Configurable and Reliable Data Storage on Hybrid Volunteered Cloud and Desktop Systems

In this paper, we propose a hybrid storage framework WukaStore, which offers a scalable, configurable and reliable big data storage service by integrating stable storage (such as Cloud storage or durable storage utilities) and volatile storage (such as idle storage harnessed from desktop PCs over the Internet). By configuring different storage strategies, WukaStore delivers cost-effective storage service to satisfy users’ requirements. We present trace-driven simulations to study the impact of different strategies and how to ensure high availability and durability in WukaStore. We present the prototype implementation of WukaStore based on BitDew, an open source middleware for Big Data management. We conduct performance evaluations to validate the effectiveness of WukaStore, through the deployment of WukaStore in the French Grid'5000 experimental platform. In particular, we also present a case study where WukaStore is deployed in a hybrid environment taking advantage of storage provided by Amazon S3, Dropbox and a local Desktop Grid at the same time. Our evaluation results show that WukaStore provides users with an alternative method to store their big data on hybrid volunteered cloud and desktop systems and to decrease the cost of data storage, while providing great data access performance, scalability as well as reliability.

Cloud Storage Service Architecture Providing the Eventually Consistent Totally Ordered Commit History of Distributed Key-Value Stores for Data Consistency Verification

Cloud storage services are one of the most popular cloud computing service types these days. Various cloud storage services such as Amazon S3, DropBox, Google Drive, and Microsoft OneDrive currently support billions of users. Nevertheless, data consistency of the underlying distributed key-value store of cloud storage services remains a serious concern, making potential customers of cloud services hesitate to migrate their data to the cloud. Researchers have explored how to allow clients to verify the behavior of untrusted cloud storage services with respect to consistency models. However, previous proposals are limited because they rely on a strongly consistent history server to provide a totally ordered history for clients. This work presents Relief, a novel cloud storage service exposing an eventually consistent totally ordered commit history of the underlying distributed key-value store to enable client-side data consistency verification for various consistency models. By empirically evaluating our system, we demonstrate that Relief is an efficient solution to overcome the limitation of previous approaches.

Effective and efficient skyline query processing over attribute-order-preserving-free encrypted data in cloud-enabled databases

Making co-existent and convergent the need for efficiency of relational query processing over Clouds and the security of data themselves is figuring-out how one of the most challenging research problems in the Big Data era. Indeed, in actual analytics-oriented engines, such as Google Analytics and Amazon S3, where key–value storage-representation and efficient-management models are employed as to cope with the simultaneous processing of billions of transactions, querying encrypted data is becoming one of the most annoying problem, which has also attracted a great deal of attention from the research community. While this issue has been applied to a large variety of data formats, e.g. relational, RDF and multidimensional data, very few initiatives have pointed-out skyline query processing over encrypted data, which is, indeed, relevant for database analytics. In order to fulfill this methodological and technological gap, in this paper we introduce an innovative algorithm for effectively and efficiently supporting skyline query processing over encrypted data in Cloud-enabled databases, named as Attribute-Order-Preserving-Free-SFS (AOPF-SFS), a suitable extension of the well-known Sort-Filter-Skyline (SFS) algorithm. The proposed algorithm enables the processing of skyline queries over encrypted data, even without preserving the order on each attribute as order-preserving encryption would do. We also present eSkyline, a prototype system that embeds AOPF-SFS equipped with a suitable query interface comprising an encryption scheme that facilitates the evaluation of domination relationships, hence allows for state-of-the-art skyline processing algorithms to be used. In order to prove the effectiveness and the reliability of our system, we also provide the details of the underlying encryption scheme, plus a suitable GUI that allows a user to interact with a server, and showcases the efficiency of computing skyline queries and decrypting the results.

The evolution of Amazon redshift

The evolution of Amazon redshift

A comparative experimental study of distributed storage engines for big spatial data processing using GeoSpark.

With increasing numbers of GPS-equipped mobile devices, we are witnessing a deluge of spatial information that needs to be effectively and efficiently managed. Even though there are several distributed spatial data processing systems such as GeoSpark (Apache Sedona), the effects of underlying storage engines have not been well studied for spatial data processing. In this paper, we evaluate the performance of various distributed storage engines for processing large-scale spatial data using GeoSpark, a state-of-the-art distributed spatial data processing system running on top of Apache Spark. For our performance evaluation, we choose three distributed storage engines having different characteristics: (1) HDFS, (2) MongoDB, and (3) Amazon S3. To conduct our experimental study on a real cloud computing environment, we utilize Amazon EMR instances (up to 6 instances) for distributed spatial data processing. For the evaluation of big spatial data processing, we generate data sets considering four kinds of various data distributions and various data sizes up to one billion point records (38.5GB raw size). Through the extensive experiments, we measure the processing time of storage engines with the following variations: (1) sharding strategies in MongoDB, (2) caching effects, (3) data distributions, (4) data set sizes, (5) the number of running executors and storage nodes, and (6) the selectivity of queries. The major points observed from the experiments are summarized as follows. (1) The overall performance of MongoDB-based GeoSpark is degraded compared to HDFS- and S3-based GeoSpark in our experimental settings. (2) The performance of MongoDB-based GeoSpark is relatively improved in large-scale data sets compared to the others. (3) HDFS- and S3-based GeoSpark are more scalable to running executors and storage nodes compared to MongoDB-based GeoSpark. (4) The sharding strategy based on the spatial proximity significantly improves the performance of MongoDB-based GeoSpark. (5) S3- and HDFS-based GeoSpark show similar performances in all the environmental settings. (6) Caching in distributed environments improves the overall performance of spatial data processing. These results can be usefully utilized in decision-making of choosing the most adequate storage engine for big spatial data processing in a target distributed environment.

Malware detection using machine learning with cloud support

This paper “malware detection using machine learning with cloud support” aims to present the functionality and accuracy of five different machine learning algorithms to detect whether an executable is infested or clean. It starts by prompting the user to login into the system by entering valid credentials. Upon successful validation of the details, the user is asked to upload the binary that he/she wants to store in the cloud database. After this, when the user uploads a file from his system, the underlying machine learning algorithms would analyse that the file is benign or infested by means of various classification algorithms. The output provided by the algorithm with highest accuracy is considered and the corresponding result is displayed to the user. The user thus becomes aware if the file uploaded by him/her contains malware or not. In case the file is found to be containing malware, it is discarded, and on the contrary if it is found to be legitimate it is stored on the Cloud by making use of Amazon S3(Simple Storage Service) and thus can be accessed by the user anytime in future. This is how malware is detected successfully by making use of Web, Machine Learning and Cloud Services.

PERFORMANCE ANALYSIS OF COVID 19 IN CLOUD COMPUTING ENVIRONMENTS

The COVID-19 pandemic has highlighted the urgent need for efficient and scalable solutions for analyzing large volumes of open research datasets. To address this challenge, we propose a cloud-based platform for mining and analyzing COVID-19 open research datasets using natural language processing (NLP), machine learning, and data visualization techniques. The proposed system comprises several components, including data ingestion, data preprocessing, text classification, in sights and visualization, and cloud computing infrastructure. Our system leverages cloud-based storage services, such as Amazon S3, and cloud computing infrastructure, such as Amazon EC2, to process large volumes of data and perform complex computations. The main contributions of this paper include a detailed description of the proposed system architecture, the data ingestion and preprocessing techniques used, the text classification algorithms employed, and the insights and visualization techniques used to generate meaningful patterns and trends in the data. We believe that our proposed system has the potential to significantly impact COVID-19 research and support the development of new treatments, vaccines, and public health policies.

A Low-Cost Multi-Failure Resilient Replication Scheme for High-Data Availability in Cloud Storage

Data availability is one of the most important performance factors in cloud storage systems. To enhance data availability, replication is a common approach to handle the machine failures. However, previously proposed replication schemes cannot effectively handle both correlated and non-correlated machine failures, especially while increasing the data availability with limited resources. The schemes for correlated machine failures must create a constant number of replicas for each data object, which often neglects diverse data popularities and does not utilize the resource to maximize the expected data availability. Also, the previous schemes neglect the consistency maintenance cost and the storage cost caused by replication. It is critical for cloud providers to maximize data availability (hence minimize SLA violations) while minimizing costs caused by replication in order to maximize the revenue. In this paper, we build a nonlinear integer programming model to maximize data availability in both types of failures, and therefore minimize the cost caused by replication. Based on the model's solution for the replication degree of each data object, we propose a low-cost multi-failure (correlated and non-correlated machine failures) resilient replication scheme (MRR). MRR can effectively handle both correlated and non-correlated machine failures, considers data popularities to enhance data availability, and also tries to minimize consistency maintenance and storage cost. Extensive numerical results from trace parameters and experiments from real-world Amazon S3 demonstrate that MRR achieves high data availability, low data loss probability and low consistency maintenance and storage costs when compared to previous replication schemes.

HyCloud: Tweaking Hybrid Cloud Storage Services for Cost-Efficient Filesystem Hosting

Today’s cloud storage infrastructures typically provide two distinct types of services for hosting files: object storage like Amazon S3 and filesystem storage like Amazon EFS. In practice, a cloud storage user often desires the advantages of both—efficient filesystem operations with a low unit storage price. An intuitive approach to achieving this goal is to combine the two types of services, e.g., by hosting large files in S3 and small files together with directory structures in EFS. Unfortunately, our benchmark experiments indicate that the clients’ download performance for large files becomes a severe system bottleneck. In this article, we attempt to address the bottleneck with little overhead by carefully tweaking the usages of S3 and EFS. Guided by two key observations, we design and implement an open-source system called HyCloud. It automatically invokes the data APIs of S3 and EFS on behalf of users, and intelligently schedules the data transfer among S3, EFS and the clients in a distributed manner. Real-world evaluations demonstrate that the unit storage price of HyCloud is close to that of S3, and the filesystem operations are executed as quickly as in EFS in most times (sometimes even more quickly than in EFS).

Delta lake

Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

With the growing amount of information and data, object-oriented storage systems have been widely used in many applications, including the Google File System, Amazon S3, Hadoop Distributed File System, and Ceph, in which load balancing of metadata plays an important role in improving the input/output performance of the entire system. Unbalanced load on the metadata server leads to a serious bottleneck problem for system performance. However, most existing metadata load balancing strategies, which are based on subtree segmentation or hashing, lack good dynamics and adaptability. In this study, we propose a metadata dynamic load balancing (MDLB) mechanism based on reinforcement learning (RL). We learn that the Q_learning algorithm and our RL-based strategy consist of three modules, i.e., the policy selection network, load balancing network, and parameter update network. Experimental results show that the proposed MDLB algorithm can adjust the load dynamically according to the performance of the metadata servers, and that it has good adaptability in the case of sudden change of data volume.

A cloud-based bioinformatic and analytic infrastructure for the Expanded Program on Immunization Consortium

Abstract The overarching goal of the Expanded Program for Immunization Consortium - Human Immunology Project Consortium (EPIC-HIPC) is to identify and characterize vaccine-induced neonatal responses and define biomarkers that may predict immunogenicity. Key to this effort is the establishment of the Data Management Core (DMC) to provide reliable clinical data and bioinformatic infrastructure for centralized curation, storage, and analysis of multiple deidentified ‘omics datasets. The DMC established a cloud-based platform to track, store, and share data according to set standards using Amazon Web Services (AWS). In our clinical core sites, biosamples collected and shipped across sites are tracked using ItemTracker via AWS Elastic Compute Cloud while their associated clinical data are captured using Research Electronic Data Capture software. Multi-omic datasets are stored in access-regulated Amazon Simple Storage Service (S3) for file version control. All data must complete quality control (QC) processes by the site generating said data, which is then exported to the DMC for quality assurance (QA). Data integration is performed using RStudio Server Pro which directly imports the data files from Amazon S3 via a controlled computing environment. The DMC deposits finalized datasets onto public repositories to be shared openly upon publication. Completion of our goal will provide the resources, planning, and scientific expertise to make this discovery platform possible. Robust DMC operations will allow rapid sharing of integrative results across the entire team. Maintenance of standards and public deposition of high quality ‘omics data will further advance scientific progress for the benefit of vaccine development and public health.

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

The improvement of Hadoop performance has received considerable attention from researchers in cloud computing fields. Most studies have focused on improving the performance of a Hadoop cluster. Notably, various parameters are required to configure Hadoop and must be adjusted to improve performance. This paper proposes a mechanism to improve Hadoop, schedule jobs, and allocate and utilize resources. Specifically, we present an improved ant colony optimization method to schedule jobs according to the job size and the time expected for execution. Priority is given to the job with the minimum data size and minimum response time. The resource usage and running jobs by data node are predicted using an artificial neural network, and job activity and resource usage are monitored using the resource manager. Moreover, we enhance the Hadoop Name node performance by adding an aggregator node to the default HDFS framework architecture. The changes involve four entities: the name node, secondary name node, aggregator nodes, and data nodes, where the aggregator node is responsible for assigning the jobs among the data node, and the Name node keeps tracking only the aggregator nodes. We test the overall scheme among Amazon EC2 and S3, and show the results of throughput and CPU response time for different data sizes. Finally, we show that the proposed approach shows significant improvement compare to native Hadoop and other approaches.

МИКРОСЕРВИСНАЯ АРХИТЕКТУРА ПРИ РЕШЕНИИ ЗАДАЧИ ПОИСКА В ОБЪЕКТНЫХ ХРАНИЛИЩАХ

This article discusses microservice architecture when solving the problem of searching object storage similar to Amazon S3. The main focus is on popular solutions and core services used in microservice architecture. Ready-made search engines are also considered: Elastissearch; Apache Solr, when solving a search problem in a developed application. Describes tools that facilitate development and debugging in a microservice architecture.

ImageBox 2 – Efficient and Rapid Access of Image Tiles from Whole-Slide Images Using Serverless HTTP Range Requests

Background: Whole-slide images (WSI) are produced by a high-resolution scanning of pathology glass slides. There are a large number of whole-slide imaging scanners, and the resulting images are frequently larger than 100,000 × 100,000 pixels which typically image 100,000 to one million cells, ranging from several hundred megabytes to many gigabytes in size. Aims and Objectives: Provide HTTP access over the web to Whole Slide Image tiles that do not have localized tiling servers but only basic HTTP access. Move all image decode and tiling functions to calling agent (ImageBox). Methods: Current software systems require tiling image servers to be installed on systems providing local disk access to these images. ImageBox2 breaks this requirement by accessing tiles from remote HTTP source via byte-level HTTP range requests. This method does not require changing the client software as the operation is relegated to the ImageBox2 server which is local (or remote) to the client and can access tiles from remote images that have no server of their own such as Amazon S3 hosted images. That is, it provides a data service [on a server that does not need to be managed], the definition of serverless execution model increasingly favored by cloud computing infrastructure. Conclusions: The specific methodology described and assessed in this report preserves normal client connection semantics by enabling cloud-friendly tiling, promoting a web of http connected whole-slide images from a wide-ranging number of sources, and providing tiling where local tiling servers would have been otherwise unavailable.

One Size Does Not Fit All: Trade-offs between Misuse Probability and Level of Sanitization for Big Data

Big data privacy has assumed importance as the cloud computing became a phenomenal success in providing a remote platform for sharing computing resources without geographical and time restrictions. However, the privacy concerns on the big data being outsourced to public cloud storage are still exist. Different anonymity or sanitization techniques came into existence for protecting big data from privacy attacks. In our prior works, we have proposed a misusability probability based metric to know the probable percentage of misusability. We additionally planned a system that suggests level of sanitization before actually applying privacy protection to big data. It was based on misusability probability. In this paper, our focus is on further evaluation of our misuse probability based sanitization of big data approach by defining an algorithm which willanalyse the trade-offs between misuse probability and level of sanitization. It throws light into the proposed framework and misusability measure besides evaluation of the framework with an empirical study. Empirical study is made in public cloud environment with Amazon EC2 (compute engine), S3 (storage service) and EMR (MapReduce framework). The experimental results revealed the dynamics of the trade-offs between them. The insights help in making well informed decisions while sanitizing big data to ensure that it is protected without losing utility required.