A debate has arisen over whether “the spirit of sport” is an appropriate criterion for determining whether a substance should be banned. In this paper, I argue that the criterion is crucial for Olympic sport because Olympism celebrates humanity, specifically human agency, so we need to preserve the degree to which athletes are personally and morally responsible for their performances. This emphasis on what I call “athlete agency” is reflected metaphysically in the structure of sport, which characteristically prescribes inefficiencies in order to create challenges, and seeks to reduce or eliminate the proportion of a performance outside an athlete’s control. Emphasizing athlete agency also prevents wealthier and more technologically developed countries from using their resources to gain an advantage in sport. Interpreted according to athlete agency, the World Anti-Doping Agency’s (WADA) “spirit of sport” can be clarified to imply that substances, techniques, and equipment that reduce athlete agency should be reduced or eliminated, while things that increase it should be encouraged.