The idea that simple visual elements such as colors and lines are associated with specific effects — for example red being warm — is intuitive to all of us. Such associations have formed a basis for the description of artworks since the 18th century and are still fundamental to discourses on art today: art historians might describe a painting where red is dominant as warm, aggressive, or lively with the tacit assumption that beholders universally associate certain forms with certain qualities. They denominate such qualities as aesthetic effects. However, is this actually the case? We tested whether and to what extent this assumption of universality (sharing of perceived qualities) is justified. For the first time, abstract artworks and single elements (lines and colors) extracted from them were used in an experiment in which participants rated these stimuli on 14 scales derived from art literature and empirical aesthetics. To test the validity of the assumption of universality we examined on which of the dimensions there is agreement, and investigated the influence of art expertise, comparing art historians with lay people. In one study and its replication, we found significantly lower agreement than expected. For the whole artworks, participants agreed on the effects of warm-cold, heavy-light, and happy-sad, but not on the 11 other dimensions. Further, we found that the image type (artwork or its constituting elements) was a major factor influencing agreement, people agreed more on the whole artwork than on single elements. Art expertise did not play a significant role and agreement was especially low on dimensions usually of interest in empirical aesthetics (e.g., like-dislike). Our results challenge the practice of interpreting artworks based on their aesthetic effects, as these effects may not be as universal as previously thought.