The aim of this study was to examine whether fantasy should be categorized as a cognitive or affective factor in empathy measurement. The categorization is not uniform in the literature, so that an empirical test using a representative sample can provide clarity. Our sample consisted of 10,303 subjects aged between 10 and 88 years. We used the German version of the Interpersonal Reactivity Index, the Saarbrücker Persönlichkeitsfragen SPF. The Fantasy Scale has a significantly higher correlation with Empathic Concern and should therefore be assumed to be an affective component of empathy. In addition, our data showed that women had significantly higher FS scores than men, which would also indicate a more affective interpretation of the scale, as women are generally considered to be more affectively empathic. In addition, we were able to demonstrate a significant decrease in Fantasy with increasing age, especially with women, which also points to an affective interpretation.