File Size Distribution Research Articles

Asymptotic behavior of a critical fluid model for bandwidth sharing with general file size distributions

This work concerns the asymptotic behavior of solutions to a critical fluid model for a data communication network, where file sizes are generally distributed and the network operates under a fair bandwidth sharing policy, chosen from the family of (weighted) α-fair policies introduced by Mo and Walrand (IEEE/ACM Trans. Netw. 8 (2000) 556–567). Solutions of the fluid model are measure-valued functions of time. Under law of large numbers scaling, Gromoll and Williams (Ann. Appl. Probab. 19 (2009) 243–280) proved that these solutions approximate dynamic solutions of a flow level model for congestion control in data communication networks, introduced by Massoulié and Roberts (Telecommun. Syst. 15 (2000) 185–201). In a recent work (Stoch. Syst. 10 (2020) 251–273), we proved stability of the strictly subcritical version of this fluid model under mild assumptions. In the current work, we study the asymptotic behavior (as time goes to infinity) of solutions of the critical fluid model, in which the nominal load on each network resource is less than or equal to its capacity and at least one resource is fully loaded. For this we introduce a new Lyapunov function, inspired by the work of Kelly and Williams (Ann. Appl. Probab. 14 (2004) 1055–1083), Mulvany, Puha and Williams (Queueing Syst. 93 (2019) 351–397) and Paganini et al. (IEEE Trans. Automat. Control 57 (2012) 579–591). Using this, under moderate conditions on the file size distributions, we prove that critical fluid model solutions converge uniformly to the set of invariant states as time goes to infinity, when started in suitable relatively compact sets. We expect that this result will play a key role in developing a diffusion approximation for the critically loaded flow level model of Massoulié and Roberts (Telecommun. Syst. 15 (2000) 185–201). Furthermore, the techniques developed here may be useful for studying other stochastic network models with resource sharing.

Heavy-Traffic Insensitive Bounds for Weighted Proportionally Fair Bandwidth Sharing Policies

We consider a connection-level model proposed by Massoulié and Roberts for bandwidth sharing among file transfer flows in a communication network. We study weighted proportionally fair sharing policies and establish explicit-form bounds on the weighted sum of the expected numbers of flows on different routes in heavy traffic. The bounds are linear in the number of critically loaded links in the network, and they hold for a class of phase-type file-size distributions; that is, the bounds are heavy-traffic insensitive to the distributions in this class. Our approach is Lyapunov drift based, which is different from the widely used diffusion approximation approach. A key technique we develop is to construct a novel inner product in the state space, which then allows us to obtain a multiplicative type of state-space collapse in steady state. Furthermore, this state-space collapse result implies the interchange of limits as a byproduct for the diffusion approximation of the unweighted proportionally fair sharing policy under phase-type file-size distributions, demonstrating the heavy-traffic insensitivity of the stationary distribution.

Performance-efficient distributed transfer and transformation of big spatial histopathology datasets in the cloud

Whole Slide Image (WSI) datasets are giga-pixel resolution, unstructured histopathology datasets that consist of extremely big files (each can be as large as multiple GBs in compressed format). These datasets have utility in a wide range of diagnostic and investigative pathology applications. However, the datasets present unique challenges: The size of the files, propriety data formats, and lack of efficient parallel data access libraries limit the scalability of these applications. Commercial clouds provide dynamic, cost-effective, scalable infrastructure to process these datasets, however, we lack the tools and algorithms that will transfer/transform them onto the cloud seamlessly, providing faster speeds and scalable formats. In this study, we present novel algorithms that transfer these datasets onto the cloud while at the same time transforming them into symmetric scalable formats. Our algorithms use intelligent file size distribution, and pipelining transfer and transformation tasks without introducing extra overhead to the underlying system. The algorithms, tested in the Amazon Web Services (AWS) cloud, outperform the widely used transfer tools and algorithms, and also outperform our previous work. The data access to the transformed datasets provides better performance compared to the related work. The transformed symmetric datasets are fed into three different analytics applications: a distributed implementation of a content-based image retrieval (CBIR) application for prostate carcinoma datasets, a deep convolutional neural network application for classification of breast cancer datasets, and to show that the algorithms can work with any spatial dataset, a Canny Edge Detection application on satellite image datasets. Although different in nature, all of the applications can easily work with our new symmetric data format and performance results show near-linear speed-ups as the number of processors increases.

How Big Are Peoples' Computer Files? File Size Distributions Among User‐managed Collections

AbstractImproving file management interfaces and optimising system performance requires current data about users' digital collections and particularly about the file size distributions of such collections. However, prior works have examined only the sizes of system files and users' work files in varied contexts, and there has been no such study since 2013; it therefore remains unclear how today's file sizes are distributed, particularly personal files, and further if distributions differ among the major operating systems or common occupations. Here we examine such differences among 49 million files in 348 user collections. We find that the average file size has grown more than ten‐fold since the mid‐2000s, though most files are still under 8 MB, and that there are demographic and technological influences in the size distributions. We discuss the implications for user interfaces, system optimisation, and PIM research.

Popularity and Size-Aware Caching With Cooperative Transmission in Hybrid Microwave/ Millimeter Wave Heterogeneous Networks

In this paper, a file popularity and size-aware (PSA) caching scheme is proposed for a hybrid heterogeneous network (HetNet) comprising a file server, macro base stations (BSs) operating in the microwave frequency bands, and pico BSs operating in the millimeter wave frequency wave bands. Different than the state-of-the-art size-weighted popularity (SWP) and popularity-based caching schemes that assume equal file size, the file PSA caching scheme utilizes knowledge of the file popularity and size distributions to maximize the network average cache hit probability and the network average success probability subject to the available cache capacity. Moreover, the maximization of the network average success probability is shown to be non-convex and dynamic programming and branch and bound algorithms are employed to circumvent its convexity. New expressions for the optimal file caching probabilities that maximize the network average success probability in noise-limited and interference-limited HetNet scenarios are provided which are evaluated using numerical techniques. The results demonstrate that the proposed file PSA caching scheme provides up to 7% gain in the network average success probability compared to the SWP-based scheme. Lastly, the combination of coded caching with cooperative transmission achieves up to 40% gain compared to a non-cooperative transmission scheme without coded caching.

Stability of a Subcritical Fluid Model for Fair Bandwidth Sharing with General File Size Distributions

This work concerns the asymptotic behavior of solutions to a (strictly) subcritical fluid model for a data communication network, where file sizes are generally distributed and the network operates under a fair bandwidth-sharing policy. Here we consider fair bandwidth-sharing policies that are a slight generalization of the [Formula: see text]-fair policies introduced by Mo and Walrand [Mo J, Walrand J (2000) Fair end-to-end window-based congestion control. IEEE/ACM Trans. Networks 8(5):556–567.]. Since the year 2000, it has been a standing problem to prove stability of the data communications network model of Massoulié and Roberts [Massoulié L, Roberts J (2000) Bandwidth sharing and admission control for elastic traffic. Telecommunication Systems 15(1):185–201.], with general file sizes and operating under fair bandwidth sharing policies, when the offered load is less than capacity (subcritical conditions). A crucial step in an approach to this problem is to prove stability of subcritical fluid model solutions. In 2012, Paganini et al. [Paganini F, Tang A, Ferragut A, Andrew LLH (2012) Network stability under alpha fair bandwidth allocation with general file size distribution. IEEE Trans. Automatic Control 57(3):579–591.] introduced a Lyapunov function for this purpose and gave an argument, assuming that fluid model solutions are sufficiently smooth in time and space that they are strong solutions of a partial differential equation and assuming that no fluid level on any route touches zero before all route levels reach zero. The aim of the current paper is to prove stability of the subcritical fluid model without these strong assumptions. Starting with a slight generalization of the Lyapunov function proposed by Paganini et al., assuming that each component of the initial state of a measure-valued fluid model solution, as well as the file size distributions, have no atoms and have finite first moments, we prove absolute continuity in time of the composition of the Lyapunov function with any subcritical fluid model solution and describe the associated density. We use this to prove that the Lyapunov function composed with such a subcritical fluid model solution converges to zero as time goes to infinity. This implies that each component of the measure-valued fluid model solution converges vaguely on [Formula: see text] to the zero measure as time goes to infinity. Under the further assumption that the file size distributions have finite pth moments for some p > 1 and that each component of the initial state of the fluid model solution has finite pth moment, it is proved that the fluid model solution reaches the measure with all components equal to the zero measure in finite time and that the time to reach this zero state has a uniform bound for all fluid model solutions having a uniform bound on the initial total mass and the pth moment of each component of the initial state. In contrast to the analysis of Paganini et al., we do not need their strong smoothness assumptions on fluid model solutions and we rigorously treat the realistic, but singular situation, where the fluid level on some routes becomes zero, whereas other route levels remain positive.

Generalization of LRU Cache Replacement Policy with Applications to Video Streaming

Caching plays a crucial role in networking systems to reduce the load on the network and is commonly employed by content delivery networks (CDNs) to improve performance. One of the commonly used mechanisms, Least Recently Used (LRU), works well for identical file sizes. However, for asymmetric file sizes, the performance deteriorates. This article proposes an adaptation to the LRU strategy, called gLRU, where the file is sub-divided into equal-sized chunks. In this strategy, a chunk of the newly requested file is added in the cache, and a chunk of the least-recently-used file is removed from the cache. Even though approximate analysis for the hit rate has been studied for LRU, the analysis does not extend to gLRU, since the metric of interest is no longer the hit rate as the cache has partial files. This article provides a novel approximation analysis for this policy where the cache may have partial file contents. The approximation approach is validated by simulations. Further, gLRU outperforms the LRU strategy for a Zipf file popularity distribution and censored Pareto file size distribution for the file download times. Video streaming applications can further use the partial cache contents to help the stall duration significantly, and the numerical results indicate significant improvements (32%) in stall duration using the gLRU strategy as compared to the LRU strategy. Furthermore, the gLRU replacement policy compares favorably to two other cache replacement policies when simulated on MSR Cambridge Traces obtained from the SNIA IOTTA repository.

Successful file transmission in mobile D2D networks with caches

A device-to-device (D2D) wireless network is considered, where user devices also have the ability to cache content. In such networks, users are mobile and communication links can be spontaneously activated and dropped depending on the users’ relative position. Receivers request files from transmitters, these files having a certain popularity and file-size distribution. In this work a new performance metric is introduced, namely the Service Success Probability, which captures the specificities of D2D networks for the Poisson Point Process case of node distribution and both the SNR and SINR coverage model. For all scenarios, the success probability depends not just on the content placement policy, but essentially on the transmitter association policy. Association can either be cache-aware or cache-unaware depending on information availability over cache inventories. We derive explicit expressions related to the defined performance metric using stochastic geometry analysis. We further extend the analysis from one-shot transmission to consecutive transmitter associations, where the receiver is allowed to profit from node mobility and gradually receive service by associating stepwise to nodes in its vicinity. Simulations support the analytical results and explain the influence of mobility, association, fading, file-size distribution and frequency reuse on the D2D performance, while providing intuition on how to appropriately cache content on mobile storage space. Of particular interest is the investigation on how different file-size distributions (Exponential, Uniform, or Heavy-Tailed) influence the performance. Finally, it is illustrated that D2D communications considerably benefit from multiple consecutive associations, which can guarantee service completion especially in high mobility environments.

Heavy-Traffic Delay Insensitivity in Connection-Level Models of Data Transfer with Proportionally Fair Bandwidth Sharing

Motivated by the stringent requirements on delay performance in data center networks, we study a connection-level model for bandwidth sharing among data transfer flows, where file sizes have phase-type distributions and proportionally fair bandwidth allocation is used. We analyze the expected number of files in steady-state by setting the steady-state drift of an appropriately chosen Lyapunov function equal to zero. We consider the heavy-traffic regime and obtain asymptotically tight bounds on the expected number of files in the system. Our results show that the expected number of files under proportionally fair bandwidth allocation is insensitive in heavy traffic to file size distributions, thus complementing the diffusion approximation result of Vlasiou et al. [20].

Fluid Models of Population and Download Progress in P2P Networks

This paper studies partial differential equation (PDE) models for the dynamics of peer-to-peer (P2P) file-sharing networks. Using as independent variables time and a fluid measure of residual work, our PDE model tracks the population profile of the P2P swarm, allowing for general file-size distributions. Focusing on the processor-sharing discipline, which we validate as an accurate model of homogeneous P2P networks, we provide a series of analytical results invoking tools of feedback control theory. We establish local stability of the equilibrium, analyze variability around this equilibrium, and compute transient response times, all of which are shown to match tightly with simulation results for a full packet-level implementation of the BitTorrent protocol. We also extend our model to heterogeneous bandwidth scenarios, and to the case of peers contributing to the system after they finish download.

A Performance Evaluation Framework for Interference-Coupled Cellular Data Networks

In regard to the continuing network densification as a part of the solution to the mobile data traffic demand explosion, managing future 5G ultra-dense networks is becoming increasingly complex. Moreover, the problem of (partly) limited capacity in time and space requires the joint treatment of spatio-temporal data traffic and intercell interference dynamics. Concerning this matter, we propose a performance evaluation framework, which is capable of estimating various cell-specific and user-specific key performance metrics considering the complex spatio-temporal interaction of traffic and interference dynamics. We provide methods for obtaining these metrics with low complexity, making the framework attractive to self-organizing network solutions for future (ultra)dense networks. We stress the framework’s broad applicability and demonstrate the effects of internal flow and external interference dynamics on network performance under various conditions. In particular, we highlight the dominance of these dynamics over the impact of the speed of the variation of intercell interference, the scheduler, the file size distribution, and fast fading.

Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency

In end-to-end data transfers, there are several factors affecting the data transfer throughput, such as the network characteristics (e.g., network bandwidth, round-trip-time, background traffic); end-system characteristics (e.g., NIC capacity, number of CPU cores and their clock rate, number of disk drives and their I/O rate); and the dataset characteristics (e.g., average file size, dataset size, file size distribution). Optimization of big data transfers over inter-cloud and intra-cloud networks is a challenging task that requires joint-consideration of all of these parameters. This optimization task becomes even more challenging when transferring datasets comprised of heterogeneous file sizes (i.e., large files and small files mixed). Previous work in this area only focuses on the end-system and network characteristics however does not provide models regarding the dataset characteristics. In this study, we analyze the effects of the three most important transfer parameters that are used to enhance data transfer throughput: pipelining,parallelism and concurrency. We provide models and guidelines to set the best values for these parameters and present two different transfer optimization algorithms that use the models developed. The tests conducted over high-speed networking and cloud testbeds show that our algorithms outperform the most popular data transfer tools like Globus Online and UDT in majority of the cases.

On Channel Failures, File Fragmentation Policies, and Heavy-Tailed Completion Times

It has been recently discovered that heavy-tailed completion times can result from protocol interaction even when file sizes are light-tailed. A key to this phenomenon is the use of a restart policy where if the file is interrupted before it is completed, it needs to restart from the beginning. In this paper, we show that fragmenting a file into pieces whose sizes are either bounded or independently chosen after each interruption guarantees light-tailed completion time as long as the file size is light-tailed; i.e., in this case, heavy-tailed completion time can only originate from heavy-tailed file sizes. If the file size is heavy-tailed, then the completion time is necessarily heavy-tailed. For this case, we show that when the file size distribution is regularly varying, then under independent or bounded fragmentation, the completion time tail distribution function is asymptotically bounded above by that of the original file size stretched by a constant factor. We then prove that if the distribution of times between interruptions has nondecreasing failure rate, the expected completion time is minimized by dividing the file into equal-sized fragments; this optimal fragment size is unique but depends on the file size. We also present a simple blind fragmentation policy where the fragment sizes are constant and independent of the file size and prove that it is asymptotically optimal. Both these policies are also shown to have desirable completion time tail behavior. Finally, we bound the error in expected completion time due to error in modeling of the failure process.

Congestion in large balanced multirate networks

In this paper, we obtain analytical approximations for various performance measures for a large fluid stochastic network that operates under a balanced fair bandwidth allocation policy. Balanced fairness results in the insensitivity of the stationary distribution of the number in the system to the precise distribution of file sizes. Balanced fairness has been shown to coincide with proportional fairness in large systems. The model we consider is that of servers operating under balanced fair rate allocations that are accessed by a large number of independent heterogeneous flows characterized by their arrival rate and general distributions of the file sizes; and a maximum service rate associated with each type of flow. The largeness of the system is parameterized by a scaling parameter that scales the arrival rates and capacity in such a way that the ratio is fixed. By exploiting a connection of the congestion probabilities with multirate Erlang loss systems, we use local limit large deviation methods to obtain accurate approximations as the scaling increases. The paper first discusses the single link case which is then extended to the case of a parking lot model that is a special case of tree networks.

Network Stability Under Alpha Fair Bandwidth Allocation With General File Size Distribution

Rate allocation among a fixed set of end-to-end connections in the Internet is carried out by congestion control, which has a well established model: it optimizes a concave network utility, a particular case of which is the alpha-fair bandwidth allocation. This paper studies the slower dynamics of connections themselves, that arrive randomly in the network and are served at the allocated rate. It has been shown that under the condition that the mean offered load at each link is less than its capacity, the resulting queueing system is stochastically stable, for the case of exponentially distributed file-sizes. The conjecture that the result holds for general file-size distributions has remained open, and is very relevant since heavy-tailed distributions are often the best models of Internet file sizes. In this paper, building on existing fluid models of the system, we use a partial differential equation to characterize the dynamics. The equation keeps track of residual file size and therefore is suitable for general file size distributions. For alpha fair bandwidth allocation, with any positive alpha parameter, a Lyapunov function is constructed with negative drift when the offered load is less than capacity. With this tool we answer the conjecture affirmatively in the fluid sense: we prove asymptotic convergence to zero of the fluid model for general file-size distributions of finite mean, and finite-time convergence for those of finite moment. In the stochastic sense, we build on recent work that relates fluid and stochastic stability subject to a certain light-tailed restriction. We further provide the supplementary fluid stability argument to establish the conjecture for this class that includes phase-type distributions. Results are supplemented by illustrative network simulations at the packet level.

File size models for shared content over the BitTorrent Peer-to-Peer network

Peer-to-Peer (P2P) traffic has increased rapidly over the past few years, with file sharing providing the main drive behind such traffic. In this work we perform a measurement study of the content shared over the popular BitTorrent P2P file sharing network. We mathematically model the file size distributions of shared files after categorizing them into Audio, Video, Archive and CD image classes. For each of these categories we look into the most popular shared file formats and investigate their file size statistics. This provides an important milestone to building a realistic simulation framework for P2P systems, and for future analytical modeling of P2P networks.

Neuropsychological constraints to human data production on a global scale

Which are the factors underlying human information production on a global level? In order to gain an insight into this question we study a corpus of 252–633 mil. publicly available data files on the Internet corresponding to an overall storage volume of 284–675 Terabytes. Analyzing the file size distribution for several distinct data types we find indications that the neuropsychological capacity of the human brain to process and record information may constitute the dominant limiting factor for the overall growth of globally stored information, with real-world economic constraints having only a negligible influence. This supposition draws support from the observation that the files size distributions follow a power law for data without a time component, like images, and a log-normal distribution for multimedia files, for which time is a defining qualia.

A study of practical deduplication

We collected file system content data from 857 desktop computers at Microsoft over a span of 4 weeks. We analyzed the data to determine the relative efficacy of data deduplication, particularly considering whole-file versus block-level elimination of redundancy. We found that whole-file deduplication achieves about three quarters of the space savings of the most aggressive block-level deduplication for storage of live file systems, and 87% of the savings for backup images. We also studied file fragmentation, finding that it is not prevalent, and updated prior file system metadata studies, finding that the distribution of file sizes continues to skew toward very large unstructured files.

Practical Adaptive User Association Policies for Wireless Systems With Dynamic Interference

We study the impact of user association policies on flow-level performance in interference-limited wireless networks. Most research in this area has used static interference models (neighboring base stations are always active) and resorted to intuitive objectives such as load balancing. In this paper, we show that this can be counterproductive in the presence of dynamic interference that couples the transmission rates to users at various base stations. We propose a methodology to optimize the performance of a class of coupled systems and apply it to study the user association problem. We show that by properly inducing load asymmetries, substantial performance gains can be achieved relative to a load-balancing policy (e.g., 15 times reduction in mean delay). We present a practical, measurement based, interference-aware association policy that infers the degree of interference-induced coupling and adapts to it. Systematic simulations establish that both our optimized static and adaptive association policies substantially outperform various dynamic policies that can, in extreme cases, even be susceptible to Braess's paradox-like phenomena, i.e., an increase in the number of base stations can lead to worse performance under greedy association policies. Furthermore, these results are robust to changes in file-size distributions, large-scale propagation parameters, and spatial load distributions.

Effective load for flow-level performance modelling of file transfers in wireless LANs

Today, a wide range of 802.11-based Wireless LANs (WLANs) have become dominant to provide wireless Internet access for file transfers. For engineering purposes, there is a need for very simple, explicit, yet accurate, models that predict the performance of WLANs under anticipated load conditions. In this context, several detailed packet-level models have been proposed, based on fixed-point equations. Despite the fact that these models generally lead to accurate performance predictions, they do not lead to simple explicit expressions for the performance of WLANs. Motivated by this, we propose a new analytic model that captures the highly complex combined dynamics and protocol overhead of the 802.11 MAC, IP, TCP and application-layer into an explicit expression for a single parameter which will be called the effective service time. Based on the effective service time, we define the effective load to describe the flow-level performance of file transfers over WLANs with an M/ G/1 Processor Sharing (PS) model. Using the M/ G/1 PS model properties we propose a simple analytic model to obtain WLAN AP buffer content distribution. Despite the fact that PS models are heavily used in modelling flow-level performance in communication networks, an extensive validation of such models has not been published in the field, or context, of WLAN. To this end, our model is validated extensively by comparing the model-based average response times against simulations. The results show that the model leads to highly accurate predictions over a wide range of parameter combinations, including light- and heavy-tailed file-size distributions and light- and heavy-load scenarios. The simplicity and accuracy of the model make the results of high practical relevance and useful for performance engineering purposes.