Video clients employ HTTP-based adaptive bitrate (ABR) algorithms to optimize users’ quality of experience (QoE). ABR algorithms adopt video quality based on the network conditions during playback. The existing state-of-the-art ABR algorithms ignore the fact that video streaming services deploy segment durations differently in different services, and HTTP clients offer distinct buffer sizes. The existing ABR algorithms use fixed control laws and are designed with predefined client/server settings. As a result, adaptation algorithms fail to achieve optimal performance across a variety of video client settings and QoE objectives. We propose a buffer- and segment-aware fuzzy-based ABR algorithm that selects video rates for future video segments based on segment duration and the client’s buffer size in addition to throughput and playback buffer level. We demonstrate that the proposed algorithm guarantees high QoE across various video player settings and video content characteristics. The proposed algorithm efficiently utilizes bandwidth in order to download high-quality video segments and to guarantee high QoE. The results from our experiments reveal that the proposed adaptation algorithm outperforms state-of-the-art algorithms, providing improvements in average video rate, QoE, and bandwidth utilization, respectively, of 5% to 18%, about 13% to 30%, and up to 45%.