The very rules of our language games contain mechanisms of disregard. Philosophy of language tends to treat speakers as peers with equal discursive authority, but this is rare in real, lived speech situations. This paper explores the mechanisms of discursive inclusion and exclusion governing our speech practices, with a special focus on the role of gender attribution in undermining women’s authority as speakers. Taking seriously the metaphor of language games, we must ask who gets in the game and whose moves can score. To do this, I develop an eclectic analysis of language games using basic inferential role theory and the concept of a semantic index, and develop the distinction between positional authority and expertise authority, which often conflict for members of oppressed groups. Introducing the concepts of master switches and sub-switches that attach to the index and change scorekeeping practices, I argue that women’s gender status conflicts with our status as authoritative speakers because sex marking in semantics functions as a master switch—“the F-switch”—on the semantic index, which, once thrown, changes the very game. An advantage of using inferentialism for understanding disregard of women’s discursive authority is that it locates the problem in the sanctioned moves, in the deontic structure of norms and practices of scorekeeping, and not primarily in the individual intentions of particular people.