Schemas and the frequency/acceptability mismatch: Corpus distribution predicts sentence judgments

1 Introduction

The so-called “frequency/acceptability mismatch”, also called the “grammaticality/frequency gap”, refers to the observation that there is no reliable correlation between the frequency of a syntactic unit and its acceptability (Bader and Häussler 2010; Kempen and Harbusch 2005). While frequent patterns are fully acceptable, the prediction fails for rare or unattested sentences with “arbitrarily low probabilities” (Manning 2003: 309). The mismatch comes in two forms: in a ceiling mismatch, two equally acceptable units come from very different frequency bands, while two low-frequent units diverge considerably in acceptability in a floor mismatch (Bader and Häussler 2010: 316). The effect has been observed for a range of syntactic and morphological constructions (e.g., Arppe and Järvikivi 2007; Bader and Häussler 2010; Bermel and Knittl 2012; Divjak 2008; Featherston 2005; Kempen and Harbusch 2005; Manning 2003).

The mismatch is interpreted very differently between theoretical frameworks. Formal models tend to attribute it to competing modular processes that operate below a certain frequency threshold (Bader and Häussler 2010; Featherston 2005). The persistence of gradience in acceptability has also afforded suggestions that grammaticality itself may be gradient as a function of constraint accumulation (Featherston 2005; Keller 2000; Sorace and Keller 2005; Wasow 2009). Methodologically, most studies in Experimental Syntax tend to represent usage in the simplest way possible by counting how often strings of words, part-of-speech tags, or constituents occur in corpora (so-called “structural frequencies”, Bader and Häussler 2010: 313).

The mismatch is more difficult to explain for usage-based models. The assumption of an intimate relationship between performance and competence implies a close relationship of usage data and experimental behavior (Bybee 2006; Langacker 1987, 1988, 2000). The central claim is that speakers build their constructional knowledge inductively by abstracting schemas over repeated exposure to sufficiently similar instances. Since each usage event is situated in complex communicative settings, and contains formal, semantic, and pragmatic information, schemas are rich in conceptual structure, but low in specific details (Langacker 2000: 4, 10). On this view, the well-formedness of a unit depends on how well it instantiates the schema. That is, if a pattern (B) instantiates schema [A] in its full specifications, (B) is said to be grammatical (or conventional) with respect to [A]. The reverse holds for ill-formedness: the more (B) deviates from relevant specifications of [A], the greater the likelihood that (B) is unacceptable or ungrammatical. As abstractions arise bottom-up over repeated exposure, grammaticality is relative to a structure’s degree of conventionality. Each utterance is categorized against previously established schemas (Langacker 1987, 1988, 2000) and its compatibility with an established schema varies along a number of lexical, morphological, syntactic, semantic, or social dimensions. In brief, acceptability is a gradient function of compatibility of an instance with a higher-order schema.

One advantage of the compatibility view is that it does not require the assumption of a frequency threshold, because compatibility with a schema should hold across all frequency bands. From this angle, the frequency/acceptability mismatch presents a problem: if compatibility is the result of exposure to repeated use, frequency makes the wrong predictions for exceedingly rare but fully acceptable expressions. This may lead to the assumption that compatibility with a schema is poorly represented by usage and/or frequency.

This conclusion is premature, because two things must be borne in mind. First, frequency of use, assumed to underlie speaker knowledge (Langacker 1987: 59), is a conceptual notion – it does not entail the empirical measure (raw) frequency. To say that repetition reinforces entrenchment neither means that speakers keep a counter of how often they hear a structure or string, let alone that they judge an expression based on this counter. Nor does it mean that repetition only pertains to strings or formal properties. Repetition also reinforces semantic and contextual knowledge of usage events in their social settings, leading to the rich conceptual schemas described above (Langacker 2000: 11). In other words, entrenchment is not linear, but complex, contingent, and multifactorial.

Second, schemas vary in specificity and complexity (cf. Figure 1). Simplexes are low in both schematicity and complexity (e.g., time, house). They instantiate their schema directly and are by definition compatible with it. Complexes such as semi-fixed expressions (e.g., for NP’s sake) or syntactic structures (e.g., the ditransitive, [NP V NP NP]) are increasingly schematic: they are abstracted over higher type frequencies and complex lexico-grammatical relationships (Langacker 1987: 25–27; Stefanowitsch and Flach 2016: 106). As the schematicity of a unit increases, so does the contingency of the contextual information (e.g., type frequency, lexical associations, subpatterns, distributional skews). Therefore, units in different areas of the complexity space require different operationalizations and measurements to capture the compatibility of its instances (see Stefanowitsch and Flach 2016 for an overview).

Figure 1:

Linguistic units by complexity and schematicity (Stefanowitsch and Flach 2016: 106).

Corpus frequency is a very good predictor of experimental behavior of monomorphemic words, which are directly compatible with their stored schema; the accuracy of prediction depends on the quality of the corpora from which the frequencies are drawn (e.g., Arnon and Snider 2010; van Heuven et al. 2014). However, the frequency/acceptability mismatch has mainly been identified for syntactic units of high complexity and schematicity. Put simply, it results from what we may call a “methodological mismatch”: if “frequency” is understood as the number of times a unit occurs in a corpus, complex contextual properties of units further to the top right are not properly captured. This problem does not disappear if the counted unit is itself abstract (e.g., part of speech tags, phrase structures, constituents, or word order patterns), because their counts are still token frequencies or something equally one-dimensional. For instance, word order patterns are often ambiguous and can instantiate any number of distinct constructional schemas, each with their own lexical associations, distributions, and functional or structural properties: the subject-control pattern in she put energy into completing the project is licensed by a different schema than the object-control pattern in she talked him into completing the project, although both have the same linear structure (or “structural frequency”). Counting word order or constituent patterns glosses over complex properties, which is often aggravated by poor precision in automatic corpus extraction (i.e., how many hits are instances of the construction).

If, however, complex measures – e.g., verb associations, type frequencies, transitional probabilities, family sizes, distributions, skews, resemblances or relatedness – are included in a way that is suitable to capture multifactorial usage properties of the higher order schema, the frequency/acceptability mismatch is substantially reduced (Arppe and Järvikivi 2007; Bermel and Knittl 2012; Divjak 2017; Gries et al. 2005; Lau et al. 2017; Wiechmann 2008). As Divjak (2017: 372) puts it, “it is not so much the case that usage frequency has problems predicting acceptability judgments at the low end of the frequency spectrum. It is rather the case that the wrong type of frequency data has been foregrounded”. It is not surprising that converging evidence from observational and experimental data is mounting once higher-order generalizations with form–meaning correspondences (Goldberg 1995) are factored in. After all, the usage-based model assumes that approaching constructional properties this way proxies speakers’ strategies in extracting schemas from repeated exposure to language in communicative settings, but not simply by keeping a (string) counter. This knowledge indeed emerges as a non-trivial influence in acceptability tasks (Divjak 2017).

The methodological mismatch (frequency of use ≠ frequency) is probably due in no small part to the ambiguity of the term frequency and what we mean when we talk about frequency data. On the one hand, frequency of use is a foundational assumption of the usage-based model (Bybee 2006; Langacker 1987). On the other hand, the term frequency is often used as a short-hand for complex corpus-derived measures that operationalize the conceptual notion frequency of use (“usage intensity”, Stefanowitsch and Flach 2016: 108) At the same time, the critique of frequency as an oversimplified measure of entrenchment pertains to its simple count-reading (e.g., Schmid 2010). Hence, a frequency/acceptability mismatch does not per se question the role of experience for linguistic knowledge, especially if it results from a methodological mismatch. Nor should it be taken to mean that experience cannot be measured in observational data: frequency of use can be operationalized in very distinct ways even for phenomena low in complexity, including – but not limited to – raw, relative, or contingent frequencies. Each of these measures has a different relationship with experimental behavior. This will be illustrated below with go/come-v as an example of a construction relatively low in complexity and schematicity: complex measures, not simple frequencies, predict the acceptability of go/come-v. Once the methodological mismatch has been reduced, there is a substantial correlation between usage data and acceptability.

It should go without saying that neither corpus analyses nor experiments measure the underlying concepts entrenchment or competence – both are theoretical notions that cannot be accessed directly. Rather, the two methods approach grammatical knowledge from different angles: while experimental tasks tap into the effects of entrenchment (or competence), corpus data proxy their causes (Stefanowitsch and Flach 2016: 121). The latter assumption is more contentious: corpora are met with substantial skepticism, since they are primarily seen as samples of linguistic output and conventionality (Schmid 2010, 2013). Yet, it is a usage-based assumption that, as samples of production, corpora also represent samples of exposure. This sample is immensely reductive, noisy, and no doubt incomplete. However, the methodological mismatch is a reminder that we need to refine the measurement tools to make appropriate use of corpus data.

The discussion below first describes schema compatibility of go/come-v (Section 2) and the construction’s distribution in corpora (Section 3). Section 4 reports the acceptability study. Section 5 compares the experimental results with the corpus data, which addresses the question to what extent experimental performance correlates with convention, if an appropriate measure of frequency of use is chosen to approximate the schema compatibility of complex units. The main argument is that while compatibility may be independent of most forms of (raw) frequency, it is highly sensitive to usage-derived properties. Therefore, paying more attention to the multidimensionality of a complex construction can reduce the frequency/acceptability mismatch and account for gradience in acceptability (Section 6). In other words, while acceptability is subject to variation, this variation is congruent with schema compatibility and usage data.

2 Schema compatibility

The go/come-v construction is suitable for an illustration of the interplay between schemas, usage, and experimental behavior on the one hand, and the frequency/acceptability and methodological mismatches on the other, for two reasons. First, go/come-v is subject to a morphological constraint (Go see the nurse! vs. *He goes sees the nurse), which has been regarded, at least implicitly, as independent of usage and/or semantics. From a cognitive, usage-based perspective, the construction is primarily subject to a semantic constraint, which affects the likelihood of occurrence in inflectional contexts for functional reasons. This also affects the acceptability of instances that violate only the semantic constraint, but satisfy the morphological constraint. Go/come-v is thus illustrative of the effects of schema (in)compatibility. Second, given a low type frequency of two, go/come-v is low in schematicity, which makes the construction accessible for the illustration of the methodological mismatch between frequency of use and the measure frequency. This section first discusses how the formal constraint on go/come-v can be accounted for in a usage-based model.

The Bare Stem Constraint (BSC) describes the phenomenon that go/come-v is grammatical in bare form, but ungrammatical if either verb is (overtly) morphologically marked (e.g., Bjorkman 2016; Jaeggli and Hyams 1993; Pullum 1990). The construction is possible only if go or come occur as non-third singular indicatives (1a), imperatives (1b), subjunctives (c), or infinitives (1d–e); it is ungrammatical with non-plain forms (third singular, preterite, participles), cf. (2):^[1]

In formal frameworks, the BSC is accounted for as the result of morphosyntactic parameter or feature operations (Bjorkman 2016; Jaeggli and Hyams 1993). By contrast, a usage-based analysis argues that the morphological constraint follows from a semantic constraint (Flach 2015, forthcoming).

What is the semantic constraint that allows I told her to go get the paper, but not *She goes gets the paper? It is true that usage-based models have an inherently difficult task to explain ungrammaticality or why something does not occur. Usage data only contain positive evidence, so an experience-based view must turn to functional motivations for structural constraints based solely on the commonalities in positive evidence (for a similar idea on island constraints, see Ambridge and Goldberg 2008).

One thing that the vast majority of go/come-v have in common is the pragmatic situations in which they occur. As the examples in (3) illustrate, go/come-v have a conspicuous preference for orders, suggestions, invitations, or recommendations. These functions are directly encoded in imperatives (3a) or subjunctives (3b), but also occur in the leftward environment: infinitive go and come are complements of lets- or why-adhortatives (3c–d), requestive matrix verbs such as ask, force, or tell (3e), or deontic (semi-)modals such as should, must, have to, or need to (3f–g):^[2]

Considerably less frequent are uses like in (4), where directive force is absent: (4a) is a general characterization, while (4b) and (4c) describe habitual actions:

In the terms of classic Speech Act Theory (Searle 1976), orders, suggestions, and recommendations in (3) are non-assertive directives and commissives, while the examples in (4) are representatives and therefore assertive. That is, the events referred to by go/come-v are prospected, not asserted (“world-to-word direction of fit”, Davies 1986; Searle 1976). The future-time implication is motivated, although not fully predictable, by the motion verbs go and come. The directive-commissive function is clear for imperatives and adhortatives. However, for infinitival go and come, the mapping between syntactic category and pragmatic context is less straightforward: they may occur in directives in (3), but also in representatives in (4). The fine-grained subdivision of infinitival go and come is motivated by phraseology and goes beyond traditional syntactic categories.^[3]

Paying attention to the leftward environment of go/come-v recognizes communicative circumstances, which are key to describe the schema and understand the functional motivation of the BSC. The approach is based on the assumption that “you shall know a construction by the company it keeps”: the constructional meaning of go/come-v can be inferred in much the same way as the meaning of words can be inferred from their collocational behavior (Firth 1957; Harris 1954). As we will see in Section 3, the directive contexts in (3) account for 86.4% of go-v and 90.3% of come-v in the Corpus of Contemporary American English (COCA; Davies 2008). This distribution is more than merely a consequence of the BSC – it is a meaningful distributional property that identifies go/come-v as a directive, non-assertive construction.

From a Construction Grammar perspective, go/come-v qualifies as a form–meaning pair on both dimensions of the original definition (Goldberg 1995: 4). Formally, the BSC is neither predictable from go or come, which are otherwise not morphologically “defective”, nor from any other construction in English. Semantically, the functional constraint is also not predictable, although it is motivated by the motion verbs that imply futurity.

Recall that schemas arise as abstractions over repeated usage events of similar instantiations. Properties of similarity pertain to form, function, semantics, and pragmatics and involve rich conceptual content. While schema (in a Langackerian sense) and construction (in a Goldbergian sense) are often used synonymously, they are understood in this context with the following difference: both directive Go get the paper! and assertive I go get the paper are instances of the same construction, but they are not equally sanctioned by the schema. As we will see below, an imperative satisfies all functional and pragmatic properties that are associated with the go/come-v schema, whereas the indicative does not. In brief, I go get the paper is less compatible with the schema.

Compatibility between an instance and the schema depends on how well the instance conforms with the schema’s specifications (Langacker 1987, 1988, 2000). While Langacker makes a distinction between full and partial sanction, sanction is ultimately gradable: partial sanction implies a greater distance from the licensing schema as more specifications are violated (see also Flach forthcoming; Langacker 2000: 12).

An instance is fully sanctioned if it involves a mandator, who requests or suggests, and a mandatee, who receives the request or suggestion. The scene is temporally bi-partite: the mandating speech act precedes the mandated state of affairs. In imperatives and adhortatives, all mappings are direct: mandator and mandatee map onto speaker and hearer, respectively, the mandating speech act coincides with the moment of speaking and the mandated event is in the future. This also holds in adhortatives, where mandator and mandatee are co-referential (Let’s go have lunch, Why don’twego have lunch?). Instances are also compatible if configurations change slightly but retain a general alignment with the schema specifications. For instance, while requestive matrix clauses can map mandator and mandatee directly onto speaker and hearer, respectively (I’m tellingyouto go see the nurse), situations can shift along the temporal axis and/or extend to third parties (I/Theytoldhimto go see the nurse). In passives, the encoding of the mandator is absent (Shewas told to go see the nurse). Usage events with imperatives (imp), adhortatives (why, lets), and requestive patterns (req) are fully compatible with the schema, because all participants and mappings are available or pragmatically understood.

Note that two instantiations of the same context may differ in compatibility. For instance, the adhortative why don’twego see a movie? is a suggestion between speaker and hearer, whereas why don’ttheygo see a movie? can also be interpreted as an enquiry about a potential or missed option by a third party. Similarly, the compatibility of (semi-)modals depends on the type of modality: deontic expressions (mod: must; semi: have to) are directives, although the deontic source is not mapped.

Sanction is only partial in situations which lack a mandator–mandatee arrangement. Future-time expressions (will, going to) are not directive, but they are non-assertive by encoding intention, imminence, or prediction. By contrast, to-complements (to.comp: it’s fun to go blow off steam), indicatives (ind: I go see my lecturer often), and do-support (do: Do you go exercise regularly?) lack all relevant configurations. That said, the leftward nominal patterns in to-complements often invite future readings of the content clause (It’s a chance to come see history). The vast majority of actually occurring indicatives are strongly biased towards futurate uses (we go clash here tonight) or non-assertive conditionals (If you go see him again,…). That is, truly assertive go/come-v of the type I go get the paper every morning are rare in corpora.

In sum, the directive environments (imp, why, lets, and req) satisfy a semantic constraint: they map all scene and participant configurations of the higher-order schema (Langacker 1988: 132). By contrast, the contexts ind, to.comp, and do violate the semantic constraint: they are extensions with increasing distance from the schema (mod and semi are somewhere in the middle). While extensions may “pass unnoticed in normal language use” (Langacker 2000:17), their compatibility is functionally compromised, which predicts lower likelihood of occurrence and lower acceptability.

As with any gradable concept, it should be borne in mind that individual cases may escape a clear classification as fully or partially sanctioned. But the categorization by syntactic environments captures the main idea: the coding of only the syntactic environment covers the corpus data remarkably well (cf. Section 3).

Compatibility relates to similarity in the following way: a commissive let’s go see a movie is similar to another commissive why don’t we go see a movie, but not by a direct comparison of form, but via a semantic-functional overlap. Although Go see the doctor! is structurally dissimilar to I told her to go see the doctor, they are similar because the participant configurations of directives are present in both. In other words, abstracted schemas maximize relevant similarities, but minimize the effects of idiosyncratic details. Most importantly, schemas may go well beyond structural or formal properties and involve rich, abstract extra-grammatical knowledge.

Let us return to the question from the beginning of this section: how can we use positive evidence to account for why inflectional go/come-v does not occur? The key is that the inflectional contexts in English (preterite, progressive, and perfect) encode representatives: *She went saw the doctor, *We were going eating there last night, or *They have come water the plants are not directive and cannot encode a mandator–mandatee configuration. They are highly improbable to occur with go/come-v for functional reasons. In other words, the morphological constraint follows the semantic constraint because of the (non-causal) correlation between the way directives are expressed in English and the English morphological paradigm.^[4] This also explains the ungrammaticality of *They have come water the plants despite the (surface) bareness of come: it can be linked directly to the inability of English perfects to express non-assertive content. This explanation of the BSC does not require an elaborate integration of the accidental form syncretism of come (which in all likelihood has nothing to do with go/come-v). Put simply, whether come.prt is “featurally inflected” is irrelevant for the usage-based explanation of the BSC.

3 Corpus distribution

From the discussion in the previous section, we can expect that schema compatibility is reflected in corpus data in two ways. First, fully compatible contexts (imp, why, lets, req) should occur with go/come-v more frequently than expected, while the reverse should hold for semantic constraint violators (semi, ind, do, to.comp). Second, there should be a continuum from imperatives (as the most compatible) to indicatives (as the least compatible), reflecting an increasing distance from the schema. We will conclude the section with a discussion of usage data with respect to the tension between “potential space” and “instantiated space” (cf. Langacker 2000).

3.2 Correspondence analysis

Correspondence Analysis (CA) is a dimension reduction method for categorical variables that aims to detect patterns in multidimensional tabular data like in Table 1.^[6] While the mathematical background of CA is beyond the scope of this paper, the conceptual idea is relatively simple (see Greenacre 2017 for an accessible introduction; for corpus applications, see Glynn 2014; Levshina 2015). In a nutshell, the Dim1 CA_SRC values in Table 1 capture the (dis)similarity of a row relative to all other rows. For instance, the row profile, or vector, of imp = [259, 374, 108, 186, 61] is more similar to why = [14, 11, 9, 9, 1] than to ind = [59, 31, 110, 97, 271], despite the differences in frequency between imp and why. This is intuitive since the higher values cluster at the beginning of the imp and why vectors, but at the end of the ind vector. In technical terms, the vectors are the syntactic environments’ coordinates in a five-dimensional space.

Since a five-dimensional space is difficult to imagine, let alone visualize, CA reduces the complexity in a matrix (like Table 1) to a few interpretable dimensions. After reduction, the first dimension explains most of the variance. For the rows, this is represented by the standard row coordinates in Table 1 (Dim 1 CA_SRC). Given their distribution across the network, the CA_SRC values best distinguish between the syntactic environments without losing too much information. Put simply, they quantify the rows’ (dis)similarities in terms of the columns, that is, they approximate the environments’ association with go/come-v.^[7] (The same logic holds for the difference between column profiles, i.e., (dis)similarity between constructions.)

CA_SRC values can be conceptualized as a continuum. This becomes more tangible if we discuss them together with the plot in Figure 2. So-called biplots are the visual representation of a high-dimensional data structure (Table 1) in a two-dimensional space (Figure 2). Since the biplot captures 88.6% of the variance (65.3% for the first dimension on the x-axis and 23.3% for the second dimension on the y-axis), we can interpret the 2D representation with reasonable confidence.

Figure 2:

CA biplot of the go/come-(and)-v network and the control sample. Dot size represents frequency and color depth represents distinctiveness (e.g., mod is frequent, but not distinctive; why is distinctive, but not frequent).

The bi-plot simultaneously shows row profiles (Syn, blue) and column profiles (Cxn, green): distributionally related categories populate similar plot regions. The closer a category member is to the center at [0,0], the less distinctive it is for the data overall (e.g., mod) and vice versa. For instance, lets is furthest from the center because its row profile deviates most markedly from all other row profiles. This can be inferred intuitively from Table 1, where lets is near-exclusive to go-(and)-v. Hence, its nearest neighbors are the go-patterns. Note, however, that numerical distances between row and column labels are not meaningful in CA. This may be counterintuitive, but the fact that lets is very far from go-v does not mean that go-v is more strongly associated with why or semi. Rather, the “outlier” position of lets reflects the fact that lets is only associated with go-(and)-v, so it is furthest from all other constructions.

Conceptually, we can think of the plot as a map or a “constructional ecology” (Taylor 2004): go/come-v occur in non-assertive environments in the right quadrants (imp, lets, why, req, why). Average corpus use, on the other hand, occurs relatively more often in assertive environments in the left quadrants (ind, do, to.comp). Overall, the plot shows two important continua from left to right along Dimension 1. First, the constructional continuum runs from average corpus use via the coordinated types to the serial types. The come-types are more “extreme”, i.e., always further right, than the corresponding go-types. Second, the directive continuum, which is more important for our current purpose, runs from assertive (do, ind) to directive environments (lets, imp), indicated by the blue line.

Let us return to the CA_SRC values in Table 1. They correspond to the syntactic environments’ values on the x-axis of the plot.^[8] In other words, even though the relationship between the rows in Table 1 is high-dimensional, it can be represented as a one-dimensional continuum (Figure 3). This representation is without a significant loss of relevant information, if the information we are interested in is the assertive–directive continuum. This continuum underlies the order of the rows in Table 1 and corroborates the discussion in Section 2.

Figure 3:

One-dimensional representation of the assertive–directive continuum (Dim1 CA_SRC).

Before we move to the judgment experiment, we briefly return to and discuss the increasingly complex ways in which frequency of use can be measured for go/come-v (cf. Section 5), although all measures are based on the same data in Table 1. The simplest is the corpus frequency (F_CRP) of the environments, which we may express as a percentage of their occurrence in the control sample (e.g., 6.1% imp, 27.1% ind). A more complex measure is the frequency of a syntactic environment in the construction (F_CXN), expressed as the average of co-occurrence with go/come-v (e.g., 31.7% imp, 4.5% ind). This is more complex than corpus frequency, because the F_CXN value of an environment depends on the F_CXN value of all other environments in the same constructional space.

The most complex measure is CA_SRC, not necessarily because of its computation, but because it reflects, simultaneously, the association between a syntactic environment and go/come-v relative to all other syntactic environments and constructional alternatives including the control sample. CA_SRC does not derive straightforwardly from – or correlate with – either F_CRP or F_CXN: the two most strongly associated directive environments lets and why are amongst the most infrequent, both in the corpus and in the construction. However, CA_SRC is frequency-related in so far as it is based on co-occurrence across a table, that is, it is contingent on the frequency of contexts in other parts of the network (and the corpus in this case). This is the underlying logic of all association measures. Frequency (counting) and association (distribution) can be two very different things, although they often correlate. Note that the increasing complexity in terms of association goes far beyond the question of choosing the right granularity (“abstractness”) of the syntactic unit being counted in a corpus (Bader and Häussler 2010; Crocker and Keller 2006).

4 Experiment

4.4 Results

The ratings are summarized in Table 2 and plotted in Figure 4. The orderings from top to bottom (table) and left to right (figure) reflect increasing schema distance. The compatible contexts imp, why, lets, and req have the highest medians and means; they also tend to be lower in variance. Expectedly, the inflectional contexts receive the lowest ratings. There is no significant difference between go and come (go-v = −0.029; come-v = 0.030; Wilcoxon p = 0.39).^[13]

Table 2:

Summary of acceptability ratings of go/come-v: Median (over raw ratings), Mean (over z-scores), and SD (over z-scores).

Syn	Example	Median	Mean	SD
imp	Go get me some water, please.	7	0.64	0.61
why	Why don’t you come see me after class?	7	0.64	0.55
lets	Let’s go cook dinner together!	7	0.65	0.52
req	Mum asked us to go help our brother.	7	0.62	0.45
semi	I’m gonna go send them an email.	6	0.34	0.67
to.comp	She likes to go play chess.	6	0.12	0.68
ind	They come fix computers for a living.	5	–0.38	0.84
pst	My sister went bought the groceries.	2	−1.18	0.60
prt	We had gone watched a movie.	2	−1.38	0.58
3rd	Helen comes speak to her parents daily.	2	−1.44	0.73

Figure 4:

Ratings (z-score) for go/come-v by syntactic environment. Scatter dots represent associated (green) and non-associated V₂ (blue); the red line shows mean ratings for all bare environments.

Three clusters emerge. The first comprises the compatible contexts imp, why, lets, and req, which are not judged notably different from each other. In the second cluster with the semantic constraint violators semi, to.comp, and ind ratings begin to drop and variation rises, especially for ind. The third cluster contains the inflectional contexts, reflecting the hard morphological constraint.

Table 3 shows the results of the LMER model with the interaction between the syntactic environment (Syn) and V₂ association (Assoc).

Table 3:

Summary of the LMER model of acceptability of bare go/come-v. Significant variable levels and interactions shown in grey.

	Estimate	SE	t-value	Pr(>|t|)
(Intercept)	0.797	0.163	4.902	0.0002 ***
Syn = why	–0.364	0.273	−1.333	0.19 ns
Syn = lets	–0.022	0.267	–0.081	0.94 ns
Syn = req	–0.292	0.231	−1.266	0.22 ns
Syn = semi	–0.621	0.184	−3.382	0.002 **
Syn = to.comp	–0.854	0.171	−4.989	8.77e–06 ***
Syn = ind	−1.445	0.199	−7.246	1.76e–08 ***
Syn = why:Assoc = yes	0.204	0.343	0.594	0.56 ns
Syn = lets:Assoc = yes	–0.184	0.341	−0.541	0.59 ns
Syn = req:Assoc = yes	0.076	0.258	0.297	0.77 ns
Syn = semi:Assoc = yes	0.522	0.214	2.440	0.026 *
Syn = to.comp:Assoc = yes	0.435	0.202	2.154	0.032 *
Syn = ind:Assoc = yes	0.670	0.256	2.617	0.012 *

Random effects:
Groups	Name	Variance	Std. Dev.
SUBJECT	(Intercept)	0	0
VERB	(Intercept)	0.076	0.276
LENGTH	(Intercept)	0.036	0.190
Residual		0.340	0.583

1. Number of obs: 959, groups: Subject 40; Verb 39, Length 5

2. Model: lmer(Z.Score ∼ Syn + Syn*Assoc + (1 | Subject) + (1 | Verb) + (1|Length)).

The values in column 2 estimate the acceptability of a syntactic environment relative to the baseline imp (the intercept at 0.797). As the estimates are negative, all other levels of Syn are, on average, judged less acceptable than imperatives. For example, a why item is judged lower by 0.364 z-score units than imp. The differences in ratings are not significant for the compatible contexts imp, why, lets, and req (p > 0.05). However, the constraint violators semi, to.comp, and ind receive significantly lower ratings (p < 0.05).

In order to compare the experimental results with corpus measures in Section 5, LMER coefficients were extracted for each level of Syn by subtracting its estimate in column 2 from the imp baseline of 0.797 (i.e., imp: 0.797; why: 0.433; lets: 0.776; req: 0.505; semi: 0.176; to.comp: −0.056; ind: −0.648).

Overall, items with an associated V₂ receive higher ratings (mn = 0.43, sd = 0.67) than with a non-associated V₂ (mn = 0.23, sd = 0.79), which is a significant difference (Wilcoxon: p < 0.001). However, as the interaction term (Syn*Assoc) shows, an associated V₂ influences the acceptability only for the constraint violators semi, to.comp, and ind. For example, a semi-modal with an associated V₂ is judged 0.522 z-score units better than a semi-modal with a non-associated V₂. There is no influence of verb association for fully compatible items (imp, why, lets, req). This effect is graphically shown in Figure 5, which aggregates the ratings for compatible and constraint violator contexts.

Figure 5:

The interaction of compatibility and verb association in acceptability judgments.

Finally, the influence of increased schema distance is consistent for the context-specific manipulation of the semantic constraint violators. Recall that half of the items for semi are deontic (semi₁: have/need to) and the other half are intention-based (semi₂: going to). Similarly, one half of to.comp and ind items describe situations with implied futurity (to.comp₁: It’s a chance to come help a friend) or motion (ind₁: I go fetch the mail regularly), while the other express habituals (to.comp₂: They like to go watch movies) or statives (ind₂: I go sleep on the couch). For semi-modals, the difference between items closer to the schema (index 1) and those further away (index 2) is not significant (semi₁ = 0.37 vs. semi₂ = 0.31; p = 0.17), but schema distance significantly affects infinitival complements (to.comp₁ = 0.33 vs. to.comp₂ = −0.09; p < 0.001) and indicatives (ind₁ = −0.23 vs. ind₂ = −0.53; p < 0.04).

In summary, the absolute morphological constraint is expectedly confirmed, although there is no sharp drop that might be suggested by a view on go/come-v solely in terms of binary (un)grammaticality. The results are broadly compatible with the informal survey where imperatives and infinitives were acceptable, while indicatives were rejected by 20% of naïve respondents on a binary choice (Pullum 1990). Crucially, acceptability depends on whether the semantic constraint is satisfied or not. While so-called soft constraints are known to be sensitive to context manipulation (Sorace and Keller 2005: 1509), the sensitivity for contextual manipulation corresponds directly, and very systematically, to schema distance. In Langacker’s terms, the extensions to semantically incompatible contexts become noticeable “when a conflict is egregious, or when small conflicts have a cumulative effect” (Langacker 2000: 17). The cumulative effect arises with diminishing directive force in the vicinity of go/come-v. This pattern is consistent with usage data on the right level of complexity, to which we now turn.

5 Comparing corpus and experimental data

The results from the judgment task confirm that acceptability is congruent with schema compatibility, depending not only on the violation of a (hard) morphological constraint, but also on the violation of a (soft) semantic constraint. This section addresses the relationship between experimental behavior and corpus distribution.

The boxplot in Figure 4 shows that acceptability is at ceiling for four contexts and drops for constraint violators. However, the ordering of the boxplots implies a non-linear relationship between equidistant contexts. The discussion of corpus distribution above showed that this may not be the case (cf. Figures 2 and 3). Hence, Figure 6 plots corpus distribution of the contexts “to scale” on the x-axis (CA_SRC), which reflects the environments’ distances quantitatively. The relationship between corpus data and acceptability judgments (LMER coefficients) is highly correlated: the closer to the schema, the higher the acceptability (r = 0.93, t = 5.74, df = 5, p < 0.01).

Figure 6:

Corpus distribution vs. acceptability ratings (lm: R² = 0.87; with 95% CI).

We now return to the other usage measures to address the relationship between the frequency/acceptability and methodological mismatches. Recall that frequency of use of go/come-v can be measured in three increasingly complex ways: (i) the frequency of the contexts in the corpus (F_CRP), (ii) their frequency in the construction (F_CXN), and (iii) their association with the construction (CA_SRC). Table 4 shows the correlation of these measures with the coefficients from the LMER model.

First, acceptability is strongly correlated with CA_SRC. Second, raw corpus frequency (F_CRP) in the final column shows the exact opposite. In other words, the high frequency of ind in a corpus (27.1%) does not “save” the context from low acceptability with go/come-v. Conversely, the lower frequency of why or lets in the corpus and the construction does not entail low acceptability of go/come-v. Third, the correlation with frequencies in the construction (F_CXN), expressed as an average over go/come-v, is moderate, but not significant. Also recall that there was no effect of construction type (Cxn;go, come) on acceptability (Section 4.4), despite the fact that go-v (13,049) is over three times more frequent than come-v (3528). Note at this juncture that the compatibility view accounts for ceiling and floor effects in the frequency/acceptability mismatch: why and lets are as acceptable as imp although they diverge very much in frequency (cf. ceiling mismatch), while lets and ind, which are similar at least in F_CXN, diverge in acceptability (cf. floor mismatch).

A final point concerns the stability of distributional measures. The CA above is based on COCA, a balanced reference corpus with five broad genres (academic, fiction, magazine, news, spoken). Reference corpora come with the drawback that they are biased towards learned and mostly written material. They are thus highly unrepresentative of the language spoken or experienced by speakers in a speech community, especially of speakers with rare or no exposure to learned material. Spoken genres in a reference corpus or specialized corpora are not immune against this problem. For instance, the spoken section in COCA contains the language of TV debates from selected US networks, which is very different to the language of telephone conversations between strangers in the SWITCHBOARD corpus. In turn, SWITCHBOARD data is not typical of face-to-face communication. This textual variability adds to the problems when measuring (raw) frequency.

However, it appears that the correlation between judgment data and (complex) corpus distribution is robust across different data types. Table 5 shows the correlation for CA_SRC and frequency measures from individual COCA genres and selected specialized corpora, for which separate CAs were calculated by the same procedure as described in Section 3.2. The slightly lower correlation coefficients for the academic genre and CHILDES may reflect the fact that these data sources are rather atypical of adult speech.

The correlation between acceptability and CA_SRC is high and robust across corpora. This is, by and large, also true for the negative correlation with corpus frequencies (F_CRP). The correlation with construction frequencies (F_CXN) is lower, unsystematic, and statistically not significant. Whether this results from the small sample size or whether it indicates that construction frequencies are an unreliable measure across corpora cannot be read from this data.

However, there are reasons to assume that distributional measures are more robust than frequencies. How often a construction occurs (pmw, F_CXN) in a corpus is context-dependent and thus sensitive to thematic or social variation. Hence, frequencies will vary considerably between corpora. What is crucial is that their distributional behavior remains stable. For instance, the strong correlation of acceptability with the conversational data in SWITCHBOARD appears unsurprising at first sight, given the informality of go/come-v. But since SWITCHBOARD contains telephone conversations between strangers, the imperative rate of go/come-v (9.1%) is much lower than in COCA (31.7%). In SWITCHBOARD, imperative go/come-v is even less frequent than indicative go/come-v (12.1%). But what is key is that directives and commissives are overall extremely low in SWITCHBOARD (imp 1.5%) and the rate of indicatives is very high (40.8%; compared to COCA’s 27.1%). In other words, the skew of go/come-v towards directive contexts remains stable in comparison to average corpus samples, despite substantial differences in frequencies. Put simply, a CA bi-plot of SWITCHBOARD data, representing the constructional ecology in telephone conversations, looks essentially identical to the COCA plot (Flach forthcoming). A similar argument holds in the other direction for CHILDES, where go/come-v is the most frequent in any of the corpora (518.3 pmw), and which has the highest rates of imperatives (40%) and the lowest rates of indicatives (2.3%). Yet, these do not automatically show the strongest correlation with judgment data (cf. Table 5), because directives and commissives are generally high in child-directed speech, i.e., go/come-v is less distinctive in child-directed speech.

In summary, there is a robust correlation between two types of performance data – corpus distribution as a proxy to speakers’ experience with a construction and acceptability as a proxy to speakers’ knowledge thereof. While the frequency of a unit or the distribution within that unit varies considerably between corpora and/or registers, distributional measures, determined in relation to other elements across a corpus, are relatively stable between data types. In other words, schema compatibility or a construction’s ecology may be much less sensitive to imbalances or fluctuations in a specific data type.

6 General discussion

This article addressed the frequency/acceptability mismatch from the perspective of the schema, a central concept in cognitive usage-based models of language. Schemas are rich conceptual knowledge structures, which speakers extract from repeated exposure to instances of the same construction in their communicative habitat. From this angle, acceptability is a function of compatibility with a licensing schema, which accounts for the acceptability even of rare or corpus-absent patterns. While acceptability may be independent of (raw) frequency, it is not independent of usage intensity (frequency of usage). The claim is that the frequency/acceptability mismatch arises in large parts from a methodological mismatch that tries to map simple measures onto complex syntactic phenomena.

Schema compatibility and its interplay with various usage-derived measures in the context of the frequency/acceptability mismatch was illustrated using the English go/come-v construction. A morphological constraint (BSC) leads to (un)grammaticality based on the presence or absence of inflection. A usage-based model explains the morphological constraint as the result of a semantic constraint: non-assertive constructional semantics make go/come-v functionally incompatible with inflectional contexts. The results of an acceptability judgment task with sentences that were specifically manipulated to reflect constructional semantics confirmed that increasingly stronger violations of schema specifications correspond with decreased acceptability. The systematicity in gradient acceptability is difficult to account for if the BSC is seen as primarily morphological.

The study illustrates that the acceptability patterns are better captured by complex than by simplistic measures. In line with recent research, the frequency/acceptability mismatch is significantly reduced with distributional measures that represent a construction’s usage properties more appropriately (Divjak 2017). Multidimensional measurements may also be considerably more robust across corpora or/and registers, which somewhat balances the extreme noisiness of corpora.

The experimental results indicate that acceptability is related to compatibility in two ways. The main influence is compatibility with the licensing schema. A minor influence emerges with respect to compatibility on a lower level: verb association positively affects acceptability, but only in cases of diminishing compatibility with the higher-order schema. Verb association, which can be seen as a tighter connection between two simpler constructions, may provide a fallback strategy that somewhat “saves” an otherwise semantically awkward structure. Put differently, non-association adds to soft constraint accumulation that affects acceptability (Langacker 2000; Sorace and Keller 2005).

It must be kept in mind that schema compatibility is a conceptual notion that may be very difficult to determine for structural units with higher type frequencies and/or multi-facetted interactions with smaller units, especially in morphological contexts (Arppe and Järvikivi 2007; Bermel et al. 2018; Divjak 2017). The identification of schema compatibility for go/come-v is straightforward, as the type frequency of two means that go/come-v is low in schematicity. The semantic-pragmatic dimension was easy to capture in just one variable (Syn). Yet, the construction illustrates the basic problem of the methodological mismatch rather well: units of varying degrees of specificity and schematicity require different measures of frequency of use (Stefanowitsch and Flach 2016: 106). Now recall that lexical frequency is an accurate predictor of experimental behavior in reaction time experiments for units that are low in schematicity (Arnon and Snider 2010; van Heuven et al. 2014). This is because units of low schematicity, such as simplexes (e.g., old, young, or time) or fixed complexes (e.g., how do you do), instantiate their own schema directly and are by definition compatible with it. (Except perhaps if they are manipulated in experimental conditions by violating selectional restrictions; manipulation decreases compatibility, because it affects connections to other units in the network.) The methodological mismatch does not arise here, at least not as quickly.

Yet, as the schematicity of a unit increases, so does the empirical complexity of the relationship between the licensing schema and its instantiation(s). Multiple interactions can be at work, such as type frequencies, distributional skews (or their absence), productivity of open slots, or interference from overlapping constructions and their interactions with lexical elements. Most of the complex morphological and syntactic units will be affected by a number of usage properties. With increased complexity, a certain degree of unexplained or “unexplainable” variance or divergence between observed and elicited data is not surprising (see, e.g., Arppe and Järvikivi 2007; Bermel et al. 2018). This is no doubt due in no small part to the reductive nature of corpora or the incomplete coverage of speakers’ experience.

In a similar vein, the preceding discussion does not imply that there are measures that are inherently well-equipped to capture multiple phenomena across all levels of schematicity and complexity. There is a lively debate on the predictive power of different association measures (e.g., Gries 2015; Schmid and Küchenhoff 2013; Wiechmann 2008). However, it is doubtful whether it is possible or even necessary to identify (sets of) metrics that perform best across phenomena, experimental tasks, or types of questions. This is because compatibility with a schema depends on a number of very complex and interrelated factors that may be highly idiosyncratic to a given pattern. An anonymous reviewer remarks that this bears the danger of post-hoc curve fitting, making the theory unfalsifiable. However, the hypothesis was not that CA is a better predictor of acceptability than raw frequencies (or any other measure, for that matter), but that simple measures are outperformed by complex measures for complex phenomena (which is falsifiable). CA was used here because it is an appropriate method to represent the relationship between two categorical variables. The results neither imply that CA is suitable for (all) other phenomena, nor that CA or other contingent measure remove the frequency/acceptability mismatch entirely. But what this article does argue is that the role of experience is underestimated, if experience is approximated by unsuitable means.

An analogy might illustrate this point. A meteorological model that predicts the weather in the Alps will be unsuitable to predict the weather at the coast, because it ignores sea water evaporation and salinity, which are less relevant in the mountains. More generally, any model that considers complex ecological conditions will be better at predicting local weather than reductive, simple models – but it will trivially never be perfect or transfer straightforwardly to different ecologies. The point in the current case is that simple frequency counts are unsuitable because they miss important local properties; but there will always be some variation that cannot be accounted for. The goal is to tease apart which forces are at work in which area of the constructional space under which conditions. The effect of V₂ association on semantic constraint violators is a result that would likely not have been detected in a (raw) frequency perspective.

Many studies which identify complex measures as better predictors of acceptability approach compatibility in a similar fashion. For instance, Gries et al. (2005) use collostructional methods to predict sentence completion in the as-predicative, where verb association (regard as) reflects compatibility better than construction frequency (see as). Divjak (2017) uses morphological transparency of low-frequency and potentially unknown verbs in Polish that-clauses. Transparency refers to the degree to which a verb is recognizably related to – and hence more compatible with – typical verbs in the pattern. An approach not unlike the current one is Dąbrowska (2008), who investigates the acceptability of context manipulation in wh-questions with long-distance dependencies (What do you think you are doing?). The prototypical template that speakers are exposed to in everyday language contains a do-auxiliary, a pronoun subject (you), and a perception verb (overwhelmingly think or say). Although Dąbrowska’s context manipulations are fully grammatical on a binary view, much like go/come-v, prototypical sentences received much higher ratings than progressive manipulations in the subject, verb, or auxiliary positions. Sentences received the lowest ratings if all of the slots deviated from the prototype, which is a form of maximal schema distance.

Now, whether gradient acceptability and/or the convergence of two types of performance data is viewed as theoretically relevant or extragrammatical will vary with one’s model of language (Sprouse et al. 2018). Depending on what one means by “grammar”, gradience will always be external to grammar (Newmeyer 2003), while some assume gradient grammaticality (Featherston 2005; Keller 2000; Wasow 2009; Weskott and Fanselow 2011). For usage-based theories, the competence–performance correspondence is more relevant, if not foundational. It crucially does not require the assumption of separate modular processes below a frequency threshold (Bader and Häussler 2010). But whether or not gradience is an integral part of one’s linguistic model of competence, the results in this study suggest that the correspondence between usage (or corpus) distribution and experimental behavior may have methodological implications. That is, complex usage properties beyond lexical or structural frequencies may need to be factored into experimental items and analyses, even if only to remove a performance effect that is irrelevant for the competence or knowledge one is interested in.