Research shows that women are more likely than men to participate in highbrow leisure activities, but we do not know whether this gap develops within the family at an early age or is the outcome of socioeconomic differences between men and women later in life. We compare highbrow leisure participation among brothers and sisters from the same family and report three findings: (1) gender differences in highbrow leisure participation are largely unrelated to family background, (2) there is little evidence that parents engage in gender-specific cultural socialization, and (3) socioeconomic position and family obligations account for less than 20% of brother-sister differences in highbrow leisure participation. Our results suggest that gender differences in highbrow leisure participation originate in factors outside the family.