Theorists have spent considerable time discussing the concept of responsibility. Their discussions, however, have generally focused on the question of who counts as responsible, and for what. But as Gary Watson has noted, “Responsibility is a triadic relationship: an individual (or group) is responsible to others for something” (Watson Agency and answerability: selected essays, 2004 , p. 7). Thus, theorizing about responsibility ought to involve theorizing not just about the actor and her conduct, but also about those the actor is responsible to—and specifically about how these people hold the actor responsible for her conduct. In this paper, I give a topology of the terrain of holding others responsible. Over the course of the paper I disambiguate two very broad senses of holding responsible—regarding another as a responsible agent and holding another responsible for a particular piece of conduct. Next, I argue that the latter sense of holding responsible is a genus with two species—what I will call “holding responsible as deep moral appraisal” and “holding responsible as accountability.” Appreciating these distinctions, I argue, sheds considerable light on a number of questions concerning the scope and nature of our practices of holding others responsible. Finally, illuminating these distinct senses of holding responsible and highlighting their features reveals an awkwardness in the most carefully explicated and influential account of holding responsible, namely R. Jay Wallace’s account in Responsibility and the Moral Sentiments.