Manufacturing cotton knitted fabric apparel generates a substantial amount of production (pre-consumer) waste. One of the ways to alleviate the environmental impact of this is to use the cotton knitted fabric production waste (CKFW) in concrete, which had the simultaneous benefit of reducing the use of virgin aggregate. The aim of this research was to examine the influence of CKFW on concrete properties, and to explore the possible application of this novel material in the construction industry for the production of precast building elements based on its characteristics. An additional goal was to enhance certain properties of concrete and determine the shortcomings of CKFW concrete. A total of 10 mixes were made and tested. CKFW was used at rates of 1.7% and 3.5% of the total mix volume. All mixtures had the same amount of cement, with three mixes having the addition of silica fume. Crushed limestone was used as the aggregate with different aggregate size combinations. Two types of polycarboxylic ether-based superplasticizer were used. The properties of the fresh mix were determined by the slump method. The ultrasonic pulse velocity, dynamic modulus of elasticity, flexural strength and compressive strength were tested on 28-day-old concrete specimens. The σ–δ diagram is also presented in this paper. We learned that the flexural strength of specimens with CKFW addition was increased by up to 38% but the compressive strength was reduced by up to 20% compared to the reference concrete mix. The CKFW mixtures had higher ductility and permeability. Additionally, silica fume had a positive effect on the concretes with a higher percentage of textiles. The percentages of textile waste used in this study affected the density of concrete in a range of 0–2%. Based on the obtained results, we offer recommendations for further tests and possible applications.