Amazon S3 Research Articles

AI-Driven Mood Analysis for Employee Wellbeing: A Proactive Approach to Enhancing Workplace Productivity

This paper explores the implementation of an AI-enabled camera system to detect faces and analyze the mood, expressions, and mental health of employees as they enter the office. The data collected is stored in an Amazon S3 bucket and used to send personalized emails within the company, offering proactive wellbeing support sessions and tips to help employees improve productivity. This approach aims to enhance employee wellbeing and organizational performance through timely and tailored interventions.

Building a Data Lake on AWS: From Data Migration to AI-Driven Insights

As organizations generate and process increasing amounts of data, building data lakes on cloud platforms like AWS has become crucial to managing large datasets efficiently. This paper outlines the key steps in constructing a scalable data lake on AWS, starting from data migration to leveraging AI for insights. It explores how AWS services like S3, Glue, and SageMaker work together to facilitate data storage, transformation, and machine learning. In addition, it highlights the importance of orchestrating data pipelines with automation tools like AWS Lambda and Apache Airflow to ensure smooth, scalable, and efficient workflows. This paper explores the end-to-end process of migrating data to AWS, constructing scalable data lakes, and leveraging AI capabilities to drive actionable insights. Through practical examples, diagrams, and pseudocode, this paper provides a comprehensive guide to implementing data lakes with AWS services such as S3, Glue, and SageMaker, highlighting key considerations around data migration, storage, processing, and analytics. The role of automation tools like AWS Lambda and Airflow in orchestrating these pipelines is also discussed. Keywords AWS, Data Lake, AI-driven Insights, Data Migration, Amazon S3, AWS Glue, Amazon SageMaker, Cloud Analytics, Data Pipeline, ETL, Machine Learning

Design of software system architecture for reading and publishing amateur literary works

The article discusses the functionality and key requirements for a web application for reading and publishing amateur literary works. Against the backdrop of current changes in the book industry caused by the development of technology and the emergence of self-publishing, the need to create platforms that provide authors with the opportunity to publish content independently is emphasized. One of the main problems that authors face when publishing is the loss of the original formatting of their works when transferring them to text editors on web platforms. This paper proposes a solution to this problem by developing a web application with the function of importing the content of text files while preserving their original formatting. The article also contains a comparative analysis of popular web-based platforms for publishing works, such as Archive of Our Own, Wattpad, and Arkush, which helped to highlight the advantages and disadvantages of each of them. In particular, the difficulties associated with storing and processing unstructured data, such as images and text files, were investigated. To solve the problem of storing such data, a comparative analysis of three approaches was conducted: storing data in relational databases, local file systems, and cloud storage. Taking into account the results of this analysis, the web application under development was chosen to integrate with the Amazon Web Services cloud provider and use Amazon S3 cloud storage, which provides scalability, reliability, and convenience in managing large amounts of unstructured data. It was also decided to use the Amazon CloudFront CDN service to improve the speed of data access and increase overall system performance. In addition, the article describes the choice of a technology stack for implementing the web application, which includes Java, Spring Framework, MySQL, ORM Hibernate for the server side and HTML, CSS, React.js for the client side. The article highlights important aspects of designing a web application architecture aimed at scalability and resilience to large amounts of data and an increase in the number of active users, and provides a basis for further system improvement.

Exploring AWS S3: In-Depth Analysis of Bucket Management

Amazon Simple Storage Service (S3) is a cornerstone in modern data management strategies with in Amazon Web Services (AWS). It is pivotal in providing scalable, durable, and accessible object storage infrastructure. A key advantage of S3 is its scalability, enabling seamless storage capacity expansion without upfront investments or provisioning delays. This flexibility empowers organizations to accommodate dynamic data growth while maintaining operational agility, which is crucial in today's rapidly evolving digital landscape. S3's exceptional durability, boasting an impressive 99.999999999% reliability, ensures steadfast data integrity and resilience against potential failures, mitigating concerns about data loss and bolstering confidence in managing mission-critical information effectively. Accessibility is another crucial aspect, with robust access controls and APIs facilitating secure data retrieval, storage, and management from any location. This accessibility fosters collaboration and innovation by enabling seamless data interaction and utilization across geographical boundaries.S3's integration capabilities with other AWS services enhance operational efficiency and optimize data analytics workflows. It is a foundational element in data analytics, facilitating data ingestion, storage, and processing to derive actionable insights from diverse datasets. Furthermore, S3's versatility extends beyond storage, supporting applications such as content delivery, backup and archiving, disaster recovery, and machine learning model training. This versatility drives innovation across industries and underscores S3's role in driving digital transformation and fostering a culture rooted in data-driven decision-making. In conclusion, Amazon S3's unparalleled scalability, durability, and accessibility make it indispensable for organizations navigating the data-centric landscape of modern business environments, underlining the imperative of harnessing its full potential for organizational success.

The Making of an Data Pipeline

This paper details the development and implementation of a data engineering pipeline designed for the extraction, transformation, and loading (ETL) of data from a web-based directory. The project involves using asynchronous web scraping techniques to gather user details from a local business directory, transforming the data into a structured format, and loading it into a storage solution. The pipeline utilises Python, the HTTPX library for asynchronous HTTP requests, BeautifulSoup for HTML parsing, and Amazon S3 for data storage. By leveraging these technologies, the pipeline demonstrates an efficient approach to handling large-scale web data extraction and processing, significantly reducing the time required to gather and organise data from multiple web pages. This paper provides insights into the architecture, implementation, and performance of the ETL pipeline, highlighting the benefits and challenges of using asynchronous programming in data engineering.

Real-Time Clickstream Analytics with Apache

The purpose of this research work is to provide an overview of setting up an Apache-based real-time clickstream data lifecycle for user behaviour analysis and marketing strategy improvement. It uses tools like Apache Kafka, Apache Spark, Amazon S3, AWS Glue Data Catalog, Hive Metastore, and Tableau to meet the challenges of data collecting, purification, and storage. The design offers rapid data processing and analysis using Spark, high-throughput and fault-tolerant data import with Kafka, and scalable storage in Amazon S3. Data retrieval, querying, and transformation are made easier by AWS Glue Data Catalogue and Hive Metastore, while Tableau offers interactive visualisations. Data management capabilities are improved by optional interaction with a data warehouse and data lake. The scalable architecture accommodates increased data quantities and user traffic, and a mathematical model derives useful insights from clickstream data.

Federated Access from DOE Labs to Distributed Storage in the EIC Era of Computing

The Electron Ion Collider (EIC) collaboration and future experiment is a unique scientific ecosystem within Nuclear Physics as the experiment starts right off as a crosscollaboration between Brookhaven National Lab (BNL) & Jefferson Lab (JLab). As a result, this muti-lab computing model tries at best to provide services accessible from anywhere by anyone who is part of the collaboration. While the computing model for the EIC is not finalized, it is anticipated that the computational and storage resources will be made accessible to a wide range of collaborators across the world. The use of federated ID seems to be a critical element to the strategy of providing such services, allowing seamless access to each lab site computing resources. However, providing Federated access to a Federated storage is not a trivial matter and has its share of technical challenges. In this contribution, we focus on the steps we took towards the deployment of a distributed object storage system that integrates with Amazon S3 and Federated ID. We will first cover for and explain the first stage storage solutions provided to the EIC during the detector design phase. Our initial test deployment consisted of Lustre storage using MinIO, hence providing an S3 interface. High Availability load balancers were added later to provide the initial scalability it lacked. Performance of that system will be shown. While this embryonic solution worked well, it had many limitations. Looking ahead, the Ceph object storage is considered a top-of-the-line solution in the storage community - since the Ceph Object Gateway is compatible with the Amazon S3 API out of the box, our next phase will use a native S3 storage. Our Ceph deployment will consist of erasure coded storage nodes to maximize storage potential along with multiple Ceph Object Gateways for redundant access. We will compare performance of our next stage implementations. Finally, we will present how to leverage OpenID Connect with the Ceph Object Gateway’s to enable Federated ID access. We hope this contribution will serve the community needs as we move forward with cross-lab collaborations and the need for Federated ID access to distributed compute facilities.

Asgard: Are NoSQL databases suitable for ephemeral data in serverless workloads?

Serverless computing platforms are becoming increasingly popular for data analytics applications due to their low management overhead and granular billing strategies. Such analytics frameworks use a Directed Acyclic Graph (DAG) structure, in which serverless functions, which are fine-grained tasks, are represented as nodes and data-dependencies between the functions are represented as edges. Passing intermediate (ephemeral) data from one function to another has been receiving attention of late, with works proposing various storage systems and methods of optimization for them. The state-of-practice method is to pass the ephemeral data through remote storage, either disk-based (e.g., Amazon S3), which is slow, or memory-based (e.g., ElastiCache Redis), which is expensive. Despite the potential of some prominent NoSQL databases, like Apache Cassandra and ScyllaDB, which utilize both memory and disk, prevailing opinions suggest they are ill-suited for ephemeral data, being tailored more for long-term storage. In our study, titled Asgard, we rigorously examine this assumption. Using Amazon Web Services (AWS) as a testbed with two popular serverless applications, we explore scenarios like fanout and varying workloads, gauging the performance benefits of configuring NoSQL databases in a DAG-aware way. Surprisingly, we found that, per end-to-end latency normalized by $ cost, Apache Cassandra's default setup surpassed Redis by up to 326% and S3 by up to 189%. When optimized with Asgard, Cassandra outdid its own default configuration by up to 47%. This underscores specific instances where NoSQL databases can outshine the current state-of-practice.

Перспективи запровадження інструментів діджиталізації облікових процесів на підприємствах України

The article provides interpretations of the concept of "digitalization" by various researchers, identifies the advantages of digitalization technologies for accounting processes, considers the main differences between automation and digitalization, provides examples of digitalization tools in accounting, highlights the results of research on the impact of digitalization tools on accounting processes, analyzes popular programs for digitizing accounting processes, which include QuickBooks, Xero, FreshBooks, Wave Accounting, and the popular cloud storages Dropbox, Google Drive, Microsoft OneDrive, Box, and Amazon S3 were explored.

Aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3.

Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present aws-s3-integrity-check, a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ∼114min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ∼935gigabytes of the Amazon S3 cloud. Our aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.

Comparing Several Encrypted Cloud Storage Platforms

Cloud services and cryptographic cloud storage systems have gained popularity in recent years due to their availability and accessibility. The present systems are nonetheless still ineffectual. They are the best since they demand a lot of trust from the user or the provider. To ensure they are not violating any End-User License Agreement (EULA) clauses, providers typically keep the ability to examine the files that have been saved, and some even keep the ability to share the data. It is simple to create a copy of every piece of data when a provider has access to go through it, which is considered abuse. A typical user would have a very difficult time proving these claims because they have no method of finding any evidence supporting such claims. Due to the growing quantity of Machine Learning (ML) performed on personal user data for either tailoring advertisements or, in more severe cases, manipulating public opinion, this issue has only gotten worse in modern times. Due to the volume of users and files kept, cloud storage services are the ideal location for getting such information, whether personal or not. To retain complete anonymity, the user could take the simple step of adding a local layer of encryption. This will prevent the cloud provider from being able to decrypt the data. The requirement for ongoing key management, which becomes more challenging as the number of keys rises, is another drawback of this. To better understand normal behaviors and pinpoint potential weaknesses, this study aims to explore and assess the security of a few well-known existing cryptographic cloud storage options. Among the vendors investigated are Microsoft Azure, Tresorit, Amazon S3, and Google Cloud. Based on documentation particular to each service, this comparison was done. However, the majority of providers frequently provide only a limited amount of information or don't go into great detail about specific ideas or procedures (for instance, security in Google Cloud), leaving room for interpretation. The authors conclude by outlining a unique approach for encrypted cloud storage that employs Cocks Identity Based Encryption (IBE) and Advanced Encryption Standard (AES)-256 Cipher Block Chaining (CBC) to limit potential abuse by alerting the user anytime a file 1 inspection takes place. Cocks IBE will be utilized as an alternate cryptographic method for access controls, and AES-256 will be used for the Initialization Vector (IV) features' encryption. Additionally, Fiat-Shamir authentication will be zero-knowledge. A system like this might be used by companies who offer services in the actual world because it would boost customer confidence.

The Story of AWS Glue

AWS Glue is Amazon's serverless data integration cloud service that makes it simple and cost effective to extract, clean, enrich, load, and organize data. Originally launched in August 2017, AWS Glue began as an extract-transform-load (ETL) service designed to relieve developers and data engineers of the undifferentiated heavy lifting needed to load databases, data warehouses, and build data lakes on Amazon S3. Since then, it has evolved to serve a larger audience including ETL specialists and data scientists, and includes a broader suite of data integration capabilities. Today, hundreds of thousands of customers use AWS Glue every month. In this paper, we describe the use cases and challenges cloud customers face in preparing data for analytics and the tenets we chose to drive Glue's design. We chose early on to focus on ease-of-use, scale, and extensibility. At its core, Glue offers serverless Apache Spark and Python engines backed by a purpose-built resource manager for fast startup and auto-scaling. In Spark, it offers a new data structure --- DynamicFrames --- for manipulating messy schema-free semi-structured data such as event logs, a variety of transformations and tooling to simplify data preparation, and a new shuffle plugin to offload to cloud storage. It also includes a Hivemetastore compatible Data Catalog with Glue crawlers to build and manage metadata, e.g. for data lakes on Amazon S3. Finally, Glue Studio is its visual interface for authoring Spark and Python-based ETL jobs. We describe the innovations that differentiate AWS Glue and drive its popularity and how it has evolved over the years.

Technical Perspective for Sherman: A Write-Optimized Distributed B+Tree Index on Disaggregated Memory

Separation of compute and storage has become the defacto standard for cloud database systems. First proposed in 2007 for database systems [2], it is now widely adopted by all major cloud providers such as Amazon Redshift, Google BigQuery, and Snowflake. Separation of compute and storage adds enormous value for the customer. Users can scale storage independently of compute, which enables them to only pay for what they really uses. Consider a scenario in which data grows linearly over time, but most queries only access the last month of data, which remains relatively stable. Without the separation of compute and storage, the user would gradually be forced to significantly increase the database cluster capacity. In contrast, modern cloud database systems allow scaling the storage separately from compute; the compute cluster stays the same over time, whereas the data is stored on cheap cloud storage services, like Amazon S3.

Privacy-Preserving Ciphertext-Policy Attribute-Based Search over Encrypted Data in Cloud Storage

Cloud storage is one of the cloud computing services which allows data users to store their data remotely to the cloud. Thus, most individuals, institutions, and organizations are outsourcing their data to the cloud. Most popular cloud-based storage services are Amazon S3, Google Drive, Microsoft Azure, Apple iCloud, Dropbox, etc. Cloud storage service brings significant benefits to data owners, say, (1) reducing capital and management costs (2) reducing cloud users’ burden of storage management and equipment maintenance, (3) avoiding investing a large amount of hardware, (4) accessing data over the Internet from any location from any devices such as desktop computers, laptops, tablets, and smartphones which offers increased flexibility and accessibility.

Real-Time SLAM Mobile Robot and Navigation Based on Cloud-Based Implementation

This study investigates the feasibility of a mobile robot navigating and discovering its location in unknown environments, followed by the creation of maps of these navigated environments for future use. First, a real mobile robot named TurtleBot3 Burger was used to achieve the simultaneous localization and mapping (SLAM) technique for a complex environment with 12 obstacles of different sizes based on the Rviz library, which is built on the robot operating system (ROS) booted in Linux. It is possible to control the robot and perform this process remotely by using an Amazon Elastic Compute Cloud (Amazon EC2) instance service. Then, the map to the Amazon Simple Storage Service (Amazon S3) cloud was uploaded. This provides a database to display maps and use them at any time for navigation without the need to redraw the map. This map can be accessed by using an authentication process (username and password) supervised by the cloud server administrator. After that, using the serverless image handler (SIH), with the aid of this solution, you can change the size of images, change the color of the background, format them, or add watermarks. Experiment results demonstrated the ability to build a map of an unknown location in a complex environment and use it for navigation tasks on a real mobile robot via remote control. It also showed the success of the process of storing the map for future use and the process of modifying the map using SIH.

Transformer vibration and noise monitoring system using internet of things

AbstractDuring continuous operation, transformer problems can occur due to various reasons. In reality, the operating parameters of the transformer have been collected and monitored through the supervisory control and data acquisition (SCADA) systems. However, these systems face many challenges when applied to no‐human substations. Currently, noise signals have been used to detect transformer errors. Abnormal noise recognition and vibration monitoring can recognize the transformer's potential defects and errors. In this study, the authors built an internet of things (IoT) system that allows remote control centres to monitor the condition of transformers through noise and vibration at non‐human substations. The proposed model was equipped with a wireless sensor network node consisting of vibration sensors, audio collectors, Arduino modules, and Lora modules. The authors set up two schemes for the IoT network: one sensor node for a 220‐kV transformer and three sensor nodes for all three phases of the 500‐kV transformer. The data obtained from the sensor node were sent to LoRa Gateway and displayed on the computer through LabVIEW. The study also enabled monitoring of parameters through IoT devices such as Desktops, Laptops, Smartphones from LabVIEW NXG Web VI platform, ThingSpeak, and Amazon S3 storage cloud. In addition, a Model Predictive Control (MPC) algorithm was applied to predict the deterioration of transformer health to maintain the system stability and, hence, prolong the transformer life and operability.

AMAZON S3 STATIC WEBSITE HOSTING

Nowadays, especially in period of energetic crisis, it is very important to get reliable data storage and uninterrupted access to websites. Considering that the world’s mainstream is using the clouds services for educational, scientific, business purposes it is important to explore features of hosting websites in major cloud service providers. One of the leader’s vendors in cloud servicing is Amazon with its Web Services (AWS). AWS offers hosting solutions such as Lightsail, Amplify Console, Simple Storage Service (S3) and Elastic Cloud Computing, from simple static websites to complicated dynamic, The article deals with the method of hosting static websites. Simple Storage Service was used for this purpose. The method has many benefits: the service is simple, cheap, scales well, manages everything and has great integration with other AWS services. This method is the best for websites that do not contain server-side scripting, like PHP or ASP.NET. It is ideal for websites that change infrequently, such as personal, promo, startup websites, websites of small businesses and organizations. The article focuses on how set permissions, properties and policies of AWS S3 bucket for hosting a website. But usage of S3 bucket alone provides only http connection. That’s why to deploy secure connection and speed up delivery of the static content was proposed to use AWS CloudFront service. CloudFront is content delivery network service, that uses the latest version of Transport Layer Security Protocol and saves files in the cache for 24 hours. The article shows how to configure CloudFront distribution to serve https requests for an Amazon S3 bucket. Configuration S3 bucket REST API endpoint was used as the most secure case. As a result, latency was reduced, security was improved due to traffic encryption and the content of S3 bucket was kept private.

RLTiering: A Cost-Driven Auto-Tiering System for Two-Tier Cloud Storage Using Deep Reinforcement Learning

The cloud storage boom has prompted providers to offer two storage tiers, i.e., hot and cold tiers, which are respectively purpose-built to provide the lowest cost for frequent and infrequent access patterns. However, for cloud users, it is non-trivial to determine cost-effective tiers because it is hard to obtain future access patterns in advance and is difficult to predict them exactly. The lack of future information poses a risk of increasing costs instead of saving costs. This is not the only challenge encountered when it comes to cost optimization. In this article, we take Amazon S3 as an example to analyze the pricing of two-tier cloud storage and derive several major challenges faced by cost optimization. Then, assuming a priori knowledge of future access patterns, we propose an optimal offline algorithm based on dynamic programming to determine cost-effective tiers for each time slot. Further, to handle online workload arrivals, we formulate the problem using Markov decision processes and propose RLTiering based on deep reinforcement learning. Eventually, the cost performance of RLTiering is evaluated based on real-world traces and prevalent Amazon S3 pricing, and the results show that it achieves significant cost-savings.

Evaluation and Implementation of Various Persistent Storage Options for CMSWEB Services in Kubernetes Infrastructure at CERN

This paper summarizes the various storage options that we implemented for the CMSWEB cluster in Kubernetes infrastructure. All CMSWEB services require storage for logs, while some services also require storage for data. We also provide a feasibility analysis of various storage options and describe the pros/cons of each technique from the perspective of the CMSWEB cluster and its users. In the end, we also propose recommendations according to the service needs. The first option is the CephFS which can be mounted multiple times across various clusters and VMs and works very well with k8s. We use it both for data and the logs. The second option is the Cinder volume. It is the block storage that runs the filesystem on top of it. It can only be attached to one instance at a time. We use this option only for the data. The third option is S3 storage. It is object storage that offers a scalable storage service that can be used by applications compatible with the Amazon S3 protocol. It is used for the logs. For S3, we explored two mechanisms. For the first scenario, we consider fluentd that runs as a sidecar container in the service pods and sends logs to S3 bucket. For the second scenario, we considered filebeat that runs as a sidecar container in the service pod and scaps those logs to fluentd which runs as a daemonset in each node and sends those logs to S3 in the end. The fourth option is EOS. We configured EOS inside the pods of the CMSWEB services. The fifth option that we explored is to use dedicated VMs that have Ceph volume attached to them. In EOS and VM, the logs from the service pods are sent to EOS/VM using the rsync approach. The last option is to send service logs to Elasticsearch. It has been implemented using fluentd that runs as a daemonset in each node. In parallel to the sending logs to S3 fluentd also sends those logs to the Elasticsearch infrastructure at CERN.

ПОРІВНЯЛЬНИЙ АНАЛІЗ ХМАРНИХ СЕРВІСІВ ДЛЯ ОБРОБКИ ГЕОІНФОРМАЦІЙНИХ ДАНИХ

The article is devoted to a comparative analysis of cloud services for processing geographic data. It describes in detail the services - Google Cloud, Amazon Web Services and Microsoft Azure - that provide tools for storing, processing and analyzing large amounts of geographic data. The article also describes the parameters of geoinformation services, the access algorithm, and examples of program code for processing satellite data. The article describes such opportunities and limitations of using cloud services as automation, security and scalability. The conclusions and recommendations for further development of geographic information systems based on cloud services are provided. Services. Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) offer a variety of geodata storage solutions. These solutions include object storage, such as Amazon S3, Azure Blob Storage, and Google Cloud Storage, as well as geospatial databases, such as Amazon RDS, Azure Cosmos DB, and Google Cloud Firestore. In addition, each of these services provides a set of services for analyzing and processing geographic information data. For example, AWS offers services such as Amazon Athena, Amazon Redshift, and AWS Glue, which allow you to run SQL queries, conduct analytics, and integrate geodata with other services. Azure offers services such as Azure SQL Database, Azure Databricks, and HDInsight, which provide capabilities for processing and analyzing geographic data. GCP also provides services such as BigQuery, Dataflow, and Dataproc, which allow you to perform analytical operations and process large amounts of geodata. Support for integration with various geo-tools is important for analysis, such as AWS, Amazon Location Service, Amazon Ground Truth, and Amazon Rekognition, which allow you to work with geodata at different levels of complexity. Azure has Azure Maps, which provides services for geocoding, routing, and visualization of geodata. GCP also offers Google Maps Platform, which provides extensive integration with geographic technologies such as routing, geocoding, and map visualization. All these processes will allow for more efficient data processing. Keywords: cloud technologies,