Traditional arguments against the identification of the language of thought with natural language assume a picture of natural language which is largely inconsistent with that suggested by contemporary linguistic theory. This has led certain philosophers and linguists to suggest that this identification is not as implausible as it once seemed. In this paper, I discuss the prospects for such an identification in light of these developments in linguistic theory. I raise a new challenge against the identification thesis: the existence of ungrammatical but acceptable expressions seems to require a gap between thought and language. I consider what must be the case in order for this objection to be dealt with.