HTTP Clients Research Articles

Streamlining PySpark Data Pipelines: Integrating External Data Sources and APIs

In the ever-expanding domain of big data analytics, integrating external data sources and APIs into data processing workflows is essential for enriching analyses and enhancing decision-making. PySpark, a prominent component of the Apache Spark ecosystem, offers robust capabilities for high-performance data processing but often requires extension to utilize external databases, web services, and third-party APIs effectively. This paper explores diverse methodologies for importing and managing data from these varied sources directly into PySpark. Techniques discussed include utilizing JDBC for database connections, employing HTTP clients for accessing RESTful APIs, and leveraging cloud storage APIs compatible with Hadoop. While integrating external data sources into PySpark pipelines significantly enriches the available data landscape, it also introduces challenges such as scalability of ingestion processes, maintenance of data integrity, and real-time data processing overhead. This study aims to provide a detailed examination of these integration techniques, highlighting both the strategic advantages and potential complications. By addressing these challenges, the paper presents a pathway towards more dynamic and comprehensive data analytics platforms, enabling businesses to leverage real-time insights and drive more nuanced analyses through PySpark.

Towards 5G: Joint Optimization of Video Segment Caching, Transcoding and Resource Allocation for Adaptive Video Streaming in a Multi-Access Edge Computing Network

Caching and transcoding at multi-access edge computing (MEC) server and wireless resource allocation in eNodeB interact with each other and together determine the quality of experience (QoE) of dynamic adaptive streaming over HTTP (DASH) clients. However, the relationship among the three factors has not been explored, which has led to limited improvement in clients’ QoE. Therefore, we propose a joint optimization framework of video segment caching and transcoding in MEC servers and resource allocation to improve the QoE of DASH clients. Based on the established framework, we develop an MEC caching management mechanism that consists of the MEC caching partition, video segment deletion, and MEC caching space transfer. Then, a joint optimization algorithm that combines the video segment caching and transcoding in the MEC server and resource allocation is proposed. In the algorithm, the clients’ channel state and the playback status and cooperation among MEC servers are employed to estimate the client's priority, video segment representation switch and continuous playback time. Considering the above four factors, we develop a utility function model of clients’ QoE. Then, we formulate a mixed-integer nonlinear programming mathematical model to maximize the total utility of DASH clients, where the video segment caching and transcoding strategy and resource allocation strategy are jointly optimized. To solve this problem, we propose a low-complexity heuristic algorithm that decomposes the original problem into multiple subproblems. The simulation results show that our proposed algorithms efficiently improve client's throughput, received video quality and hit ratio of video segments while decreasing the playback rebuffering time, video segment representation switch and system backhaul traffic.

Fuzzy-Based Quality Adaptation Algorithm for Improving QoE from MPEG-DASH Video

Video clients employ HTTP-based adaptive bitrate (ABR) algorithms to optimize users’ quality of experience (QoE). ABR algorithms adopt video quality based on the network conditions during playback. The existing state-of-the-art ABR algorithms ignore the fact that video streaming services deploy segment durations differently in different services, and HTTP clients offer distinct buffer sizes. The existing ABR algorithms use fixed control laws and are designed with predefined client/server settings. As a result, adaptation algorithms fail to achieve optimal performance across a variety of video client settings and QoE objectives. We propose a buffer- and segment-aware fuzzy-based ABR algorithm that selects video rates for future video segments based on segment duration and the client’s buffer size in addition to throughput and playback buffer level. We demonstrate that the proposed algorithm guarantees high QoE across various video player settings and video content characteristics. The proposed algorithm efficiently utilizes bandwidth in order to download high-quality video segments and to guarantee high QoE. The results from our experiments reveal that the proposed adaptation algorithm outperforms state-of-the-art algorithms, providing improvements in average video rate, QoE, and bandwidth utilization, respectively, of 5% to 18%, about 13% to 30%, and up to 45%.

Playback continuity and video quality driven optimisation for dynamic adaptive streaming over HTTP clients over wireless networks

In this study, the authors focus on segment characteristic analysis and joint optimisation of modulation and coding scheme (MCS) selection, resource block (RB) assignment and the block error ratio (BLER) determination and segment adaptation scheme to satisfy dynamic adaptive streaming over HTTP (DASH) clients over wireless networks. First, the authors define a utility function as the continuous playback time that the packets scheduled with the allocated resource can support. Then, the authors formulate the MCS selection, RB assignment and BLER determination into a mathematical model. The relationship among the above three factors can be explored instead of performing MCS selection and RB assignment with the fixed BLER. By decomposing the original problem into some sub-problems, the authors can get a solution to the original problem with low complexity. At the client level, the authors develop an adaptive segment request strategy based on the playback information, the segments' characteristics and the estimated transmission rate. To decrease the influence of the inaccurate estimations, an adaptive guard time interval based on the real playback information and transmission information of the previous segments is introduced. Simulation results show that the proposed algorithm can efficiently improve playback continuity and video quality over the existing algorithms.

SABA: segment and buffer aware rate adaptation algorithm for streaming over HTTP

Adaptive streaming allows for dynamic adaptation of the bitrate to varying network conditions, to guarantee the best user experience. Adaptive bitrate algorithms face a significant challenge in correctly estimating the throughput, as the throughput varies widely over time. The current throughput estimation methods cannot distinguish between throughput fluctuations of different amplitude and frequency. In this paper, we propose a throughput estimation method that accurately estimates the throughput based on previous throughput samples. It is robust to short term and small fluctuations, and sensitive to large fluctuations in throughput. Furthermore, we propose a rate adaptive algorithm for video on demand (VoD) services that selects the quality of the video based on the estimated throughput and playback buffer occupancy. The objective of the rate adaptive algorithms is to guarantee high video quality to improve user experience. The proposed algorithm dynamically adjusts the quality level of the video stream. The proposed method selects high quality video segments, while minimizing the risk of playback interruption. Furthermore, the proposed method minimizes the frequency of video rate changes. We show that the algorithm smoothly switches the video rate to improve user experience. Furthermore, we determine that it efficiently utilizes network resources to achieve a high video rate; competing HTTP clients achieve equitable video rates. We also confirm that variations in the playback buffer size and segment duration do not affect the performance of the proposed algorithm.

Compete or Collaborate: Architectures for Collaborative DASH Video Over Future Networks

Dynamic adaptive streaming over HTTP (DASH) clients compete with each other over one or more bottleneck links in a network, which results in fluctuations in TCP throughput and QoE, QoE unfairness among clients, and underutilization of the network capacity. We propose centralized and distributed architectures for collaboration between network service provider (NSP), video service provider (VSP), and users (DASH clients) to provide NSP-managed or VSP-managed DASH services over software-defined networks (SDN) with quality-of-service (QoS) reserved network slices. We show that QoS reservation alone is not sufficient to overcome QoE fluctuations per client and unfairness between heterogeneous video clients, and clients also need to employ TCP receive-window adaptation knowing their fair-share bitrate. To this effect, we propose two collaborative streaming service models to inform clients about their fair-share bitrates. We first present an NSP-managed service model with centralized collaboration between the NSP, VSP, and the users, where a traffic engineering manager at the NSP assigns a fair-share bitrate to each DASH client. We then present a VSP-managed service model with centralized or distributed collaboration architectures, where in the former the VSP determines the fair-share bitrate for each client over a reserved network slice and in the latter a group of DASH clients sharing a reserved network slice collaborate among themselves. In the novel distributed collaboration framework, collaboration groups are identified by the VSP, and clients within a group share critical parameters with each other so that each client can estimate its fair-share bitrate. Experimental results demonstrate that collaboration rather than competition between clients not only helps them achieve a smooth goodput near their fair-share bitrate, but also improves the total goodput over the reserved slice.

Composition of web-services using Prolog

The possibility of web-service creation with the help of pure Prolog (without other programming languages) is reviewed. The main features of web-documents parsing and generating by SWIProlog are described. The different models of HTTP servers and clients supported by SWI-Prolog are presented.

A Cross Layer Collaborating Cache Scheme to Improve Performance of HTTP Clients in MANETs

This paper proposes a novel cross layer collaborating cache scheme for HTTP clients. It is designed to accelerate the process that an HTTP client retrieves Web content in mobile ad hoc network environment. Unlike other techniques to exploit Web proxy caching or to build P2P protocol overlay to improve HTTP, our scheme incorporates collaboratively working capability into the HTTP client response cache, and thus there is no impact on the compatibility of upper level applications. In addition, as the sharing file information is diffused with routing information, there is no redundant traffic to maintain the cooperating cache mechanism. Using this scheme, cacheable Web content can be distributed among local peers and retrieved later by a client if needed. In our simulation results, this novel cache scheme significantly decreases the latency experienced by a client to download Web content.

A novel distributed covert channel in HTTP

ABSTRACTIn this paper, we propose a novel distributed covert channel in HTTP. Different from traditional covert channels in HTTP, the channel deploys multiple HTTP clients to dilute steganographic features. In a proper multiple‐to‐multiple transmission model based on the modulation of URLs, the channel is efficient and reliable. With a Poisson request generator, simulating behaviors of normal HTTP visitors, the covert traffic has a legitimate HTTP appearance that helps it be undetectable. The transmission experiments prove that the channel is error free and has a high transmission rate. The results of the undetectability experiment show that the channel is adjustable. By adjusting a certain parameter, the channel can trade off two different features, the transmission rate and the undetectability, which can meet different demands in practical applications. Copyright © 2013 John Wiley & Sons, Ltd.