Introduction

Recently, Bu et al. (2019) introduced the notion of a multi-dimensional perspective on citation impact which goes beyond simple citation counts, the usual one-dimensional perspective. The authors argue for considering other aspects, which can be derived from citation data, for assessing different kinds of research impact. There are various possible ways to improve citation analyses in the direction of more detailed meaning. Bu et al. (2019) propose new indicators based on co-citations and bibliographic coupling measuring depth and dependence of citation impact. Another approach is to label ideas, concepts, methods etc. from certain publications as “discoveries” by citing researchers (Small et al. 2017) which is not visible by simple citation counting. An obvious way of improving citation impact measurements alongside the normative view of citations (Merton 1973) is to count citations of publications with indications of the cognitive influence.

In this study, we introduce a new form of citation analysis, which we call citation concept analysis (CCA). CCA is intended to reveal the cognitive impact certain concepts have on the citing authors. It counts the number of times the concepts are mentioned (cited) in the citation context of citing publications. It only makes sense to employ CCA with publications introducing at least one important concept, which can be detected reliably in the citation context of citing publications. These publications are usually highly-cited landmark publications introducing useful concepts. The method cannot be used to measure the impact of conventional publications, those that do not introduce an important concept and that do not receive many citations. The first step of the CCA is to compile the citation context information of a certain publication based on its citing publications. In the second step, the citation context information is searched for the existence of certain concepts from the landmark publication. The third step focusses on the statistical analyses of the search results.

In this study, we explain and demonstrate the use of CCA by using different editions of several publications, each of which introduced several influential concepts: The structure of scientific revolutions by Kuhn (1962), The logic of scientific discovery (Popper 1959)—Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft (Popper 1934) in German—and Conjectures and refutations: the growth of scientific knowledge by Popper (1962).Footnote 1 The books present, explain, and employ several concepts each, e.g., “scientific revolution” or “falsification”, which can be searched in the citation context of citing publications. We are interested in how useful the concepts from the books are for the citing authors. Since the books represent two contrary understandings of scientific progress, CCA results might also reveal how scientific progress is differently interpreted in various disciplines and time periods.

In a second part of the empirical analysis, following the proposal by Small et al. (2019) and Small (2018), we searched for hedging word strings expressing uncertainty (e.g., “may” and “could”) in citation context data of Kuhn (1962) and Popper (1934, 1959, 1962). We are interested in determining which concepts are more frequently associated with uncertainty: which concepts are interpreted as scientifically proven and which not? Small et al. (2019) interpret hedging as an expression of uncertainty. The authors propose to use certain hedging words to measure uncertainty associated with cited publications. Small (2018) also reports that the hedging word “may” is more frequently observed in citation contexts for lower cited papers than for more highly-cited papers.

Theoretical reflections on citation concept analysis (CCA)

Since the use of citations in measuring research gained a certain level of attention, researchers have started to develop theories of the citation process. According to Sandström (2014), these theories try “to explain why author x cite[s] article a at time t? What factors should be considered when we discuss why researchers cite back to former literature?” (p. 59). Tahamtan and Bornmann (2018) published a conceptual overview of the literature dealing with the process of citing documents. They identified three core elements in this process: (i) the context of the cited document, (ii) processes from selection to citation of documents, and (iii) the context of the citing document. Tahamtan and Bornmann (2018) not only considered empirical studies on the citing process, but also theories of the citing process which have been proposed hitherto. Two of these theories (the first and third) are of special importance for the CCA (see below).

Robert K. Merton proposed the first theory of citing: the normative theory. Merton (1973) discusses citations in the context of various norms of science, which are part of the ethos of science. One of these norms “communism” dictates that researchers credit the original research of other researchers, and citations are a formal process in publishing research to do that. Since citations point to published research which has been highlighted by publishing researchers in their reference lists, Merton (1973) sees citations as a part of the reward system of science, which is one of the reasons why citations can be used for research evaluation purposes. The second theory of citing is in sharp contrast to the normative theory. Social constructivists (e.g., Gilbert 1977) claim that researchers cite publications for rhetorical and strategic reasons, that is, reasons that do not, or scarcely, depend on the quality of the cited publication (Case and Higgins 2000). For example, in view of the social constructivist theory of citing, researchers cite strategically certain publications which support their own claims, (and avoid citing publications with contrary results; Teplitskiy et al. 2018). Social constructivists doubt that citation decisions can be explained by only one dominant reason, as proposed by the normative theory of citing. Several reasons exist—also for single citation decisions—which are rarely related to the quality of the cited research.

Small (1978) proposed a third view of citing which can be connected to both the normative and social constructivist view. Small (1978) interprets citations as an occurrence of symbol usage. In the analysis of highly cited documents in chemistry, Small (1978) found that certain documents (important or highly-cited documents) may be seen as “‘standard symbols’ for particular ideas, methods, and experimental data” (p. 327). These concept symbols can be used by the citing authors to credit original research, following the normative theory of citing, and for rhetorical and strategic reasons, following the constructivist theory of citing. For example, Kuhn (1962) can be cited because the citing author uses the paradigm concept in their own research. Alternatively—in the spirit of the constructivist theory—would be to cite Kuhn (1962) because the citing authors believe that the reference to Kuhn (1962) has positive effects in the later assessment of their manuscript, e.g., because reviewers or editors are proponents of concepts introduced by Kuhn (1962).

The theoretical roots of the CCA are located in the first and third view of the citing process proposed by Small (1978). In the first step of the CCA, certain concepts (concept symbols) are identified which are connected with a certain (landmark) publication. In the second step, search terms are defined which can be used to identify the concepts in so-called citances. A citance is defined as a “single sentence in which the citation was made” (Small 2018, p. 463). In the third step, the statistical analysis, the search terms are used to determine the number of concept occurrences in citances. Following Merton (1973), the CCA measures cognitive influence. Small (1978) already proposed to investigate the text around citations to identify the particular idea which connects the citing authors with the cited document. The availability of large amounts of citation context data in bibliometric databases such as Microsoft Academic (MA) now makes it possible to investigate concept symbols in large datasets.

Importance of the books published by Thomas S. Kuhn and Karl R. Popper used as examples in this study

The importance of The structure of scientific revolutions by Kuhn, The logic of scientific discovery—Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft in German—, and Conjectures and refutation: the growth of scientific knowledge by Popper cannot be overstated. Popper’s Logic of scientific discovery marked the beginning of his academic career. The book was completed when he was working as a high school teacher in Austria in the 1930s. But it played an integral role in securing his first position at a university in New Zealand. In the book, Popper (1959) presents the central or organising idea of his whole career, falsificationism. He argues that, contrary to the reigning orthodoxy, scientists cannot confirm hypotheses or theories by testing them. Rather, from a logical point of view, when they test hypotheses and theories by deriving predictions from them, all they can do is attempt to falsify them. If a theory is in fact falsified, if it entails a false prediction, then it should be discarded or modified. If it is not falsified, if the predictions from it are corroborated, then it can continue to be accepted. But the acceptance of a hypothesis or theory, Popper insists, is always tentative. Some future test may prove the theory false. One further important reason why Popper’s book is regarded as significant is the role it played in challenging the then-orthodox view, Logical Positivism. The Positivists were based principally in Vienna, and it was in that milieu that Popper developed his philosophy of science. Popper would later claim that it was he who killed positivism (Popper 1992, § 17).

All the other concepts we examine from Popper’s books are intimately connected to falsificationism: verification, induction, demarcation, and probability. The positivists were seeking to develop a logic of verification. Popper, though, argued that it is not possible to verify a hypothesis or theory. Popper argued that when one infers the truth of a theory from an observation predicted from that theory, one is committing a logical fallacy, the fallacy of affirming the consequent. The positivists also aimed to develop a logic of induction. Popper, though, argued that there is no logic of induction. The only way that scientists can prove a claim is by deduction. Specifically, when a scientist derives a prediction from a theory, and the world is not as the theory predicts, then the scientist can deductively infer that the theory is false. This is an instance of the inference pattern modus tollens. Demarcation is another important concept for Popper. He believes that falsifiability is the feature that distinguishes scientific theories, like Isaac Newton’s and Albert Einstein’s, from pseudo-scientific theories, like Sigmund Freud’s theory of psychology or Karl Marx’s theory of history. Falsifiability is thus Popper’s demarcation principle. Finally, probability is another concept central to Popper’s thinking. His studies of probability were motivated by his concern to develop an understanding of corroboration. Even though vindicated predictions could not prove a theory is true, they did corroborate it, that is, offer some sort of support for it. Popper’s studies of probabilities were motivated, in part, by a concern to bring rigour to the notion of corroboration.

Kuhn’s (1962) Structure of Scientific Revolutions is also undeniably an important contribution to the philosophy of science. It was responsible for two of the citation peaks in the philosophy of science from 1900 to 1970 (see Wray and Bornmann 2015). And its effects were wide ranging (see Wray 2017). Kuhn (1962) played a crucial role in developing the successor project in the philosophy of science to the then-dominant view, Logical Positivism. Kuhn (1962) argued that the development of scientific fields was not a steady progression, but involved the periodic interruption of radical changes of theory, what he called paradigm changes. These changes involved significant changes to the basic assumptions about the nature of the world. Kuhn (1962) thus developed a cyclical account of the growth of science. On the one hand, his book was quite quickly greeted with enthusiasm as it seemed to offer insight into how science really works, drawing on a study of the history of science. On the other hand, it seemed to undermine the presumed rationality of science. Kuhn (1962) compared paradigm changes to religious conversions and gestalt shifts, processes that are typically regarded as non-rational. He also spoke of scientific fields as in crises. And competing theories were described as incommensurable. These were hardly comforting words to scientists or philosophers of science. But they certainly created excitement across the academic world. Even in literary theory students were encouraged to read Kuhn’s The structure of scientific revolutions.

The other concepts we examine in Kuhn’s Structure are related to the notion of paradigm change. They include normal science, theory change, and anomaly. Normal science is the phase of scientific research that falls between paradigm changes. In fact, normal science is the sort of research that most scientists engage in most of their careers. It involves the uncritical application or extension of an accepted paradigm or theory. And theory change is just an alternative term used to describe paradigm changes. Periods of normal science are punctuated by changes of theory. And anomalies are the problematic observations that scientists are unable to account for given the resources of the accepted theory. They are thus the cause of revolutionary changes of theory.

Literature review (of citation context and content studies)

Since the citation process is scarcely standardized (Cronin 1982), the adoption of a multi-dimensional perspective on citations seems reasonable. As outlined by Tahamtan and Bornmann (2018), several reasons and decision rules exist for citing. Consequently, it is not clear what is being measured with citations: cognitive influence, citation circles, Matthew effects, field-specific impact patterns, rhetorical persuasion etc. The results of the empirical study by Judge et al. (2007) based on 21 management journals reveals that “universalistic, particularistic, and mixed universalistic-particularistic characteristics all play significant roles in the extent to which research articles in the field of management are cited” (p. 500). Already at the end of the 1980s, based on various citation studies, Cano (1989) concluded that “neither citation behavior, nor citation use or citation impact has been satisfactorily explained” (p. 284). We think that the situation has not changed substantially since then despite considerable research efforts in this area.

A decade ago, Bornmann and Daniel (2008b) reported the results of the previous empirical literature dealing with the multi-dimensional perspective on citation impact (see also Tahamtan and Bornmann 2019). The previous studies can be assigned to three groups: (1) Citation context studies are based on a text analysis of one or two sentences around an in-text citation and aim to classify the citation (e.g., as “confirmative” or “negational”). According to Boyack et al. (2018) an in-text citation “is a mention of a reference within the full text of a document. A reference can be mentioned one or more times in a document. Each mention is an in-text citation” (p. 60). (2) Citation content analyses investigate the semantic content of the citing papers to characterize the cited publication. (3) Citer motivation interviews or surveys identify citer motives by surveying or interviewing citing authors. Following this classification, the empirical parts of the current study are citation context analyses since the text around the in-text citation is searched for the existence of certain search terms or concepts.

In recent years, the situation changed for conducting citation context or content analyses. In the past, it was very time-consuming and resource intensive to conduct these studies. The analyses had to be done manually by searching for certain in-text citations and categorizing their content. A good example of these comprehensive studies is the citation context study undertaken by Bornmann and Daniel (2008a). However, the situation changed with the growing dissemination of large datasets as outlined by Bertin et al. (2016). Due to the increasing availability of datasets, an increasing number of studies have been published using, e.g., datasets from PubMed Central, Public Library of Science (PLOS), MA, and Elsevier.

The study by Small et al. (2017) used a similar approach as we do in this current study. They searched in citing sentences for the truncated string “*discover*” to determine whether the cited paper is a revolutionary work leading to a discovery. However, the results of the study point out that this approach to citation context analysis cannot be used to reliably find discoveries: “While we have found that the appearance of ‘discovery words’ is not a reliable indicator of whether a given citance describes a discovery (the success rate is about 46%), it is relatively easy to differentiate these manually by inspecting a sample of citances. Because the majority of false hits are methodologies and tools for making discoveries, rather than actual scientific discoveries, the vocabulary used in citances in such cases has been shown to be highly instrumental, and to differ markedly from the vocabulary used when actual discoveries are cited” (p. 56).

Three studies have used various approaches for studying the use of content from cited papers more deeply. Greenberg (2009) investigated how a belief system evolves in a scientific community, specifically “the belief that β amyloid, a protein accumulated in the brain in Alzheimer’s disease, is produced by and injures skeletal muscle of patients with inclusion body myositis”. Greenberg (2009) searched for statements addressing this belief in papers indexed in PubMed related to the β amyloid. Citation networks were analyzed by social network and graph theory: “the network contained 242 papers and 675 citations addressing the belief, with 220,553 citation paths supporting it. Unfounded authority was established by citation bias against papers that refuted or weakened the belief; amplification, the marked expansion of the belief system by papers presenting no data addressing it; and forms of invention such as the conversion of hypothesis into fact through citation alone” (Greenberg 2009).

Hammarfelt (2011) investigated the impact of a highly cited publication in humanities, Walter Benjamin’s Illuminations (Benjamin 1968). Hammarfelt (2011) used a new method for measuring detailed impact by additionally considering the sections and cited pages. The results show that “a majority of the citations refer to a few specific essays in the collection ‘The Work of Art in the Age of Mechanical Reproduction,’ ‘The Storyteller,’ and ‘Theses on the Philosophy of History.’ The single most cited page is page 217, with 327 citations, which is the first page of ‘The Work of Art in the Age of Mechanical Reproduction’” (p. 826). Solomona, Youtieb, Carleyc, and Porter (2019) explored the citations the seminal US National Academies consensus report, How People Learn (HPL), received (National Research Council 2000). When they analyzed the actual content in HPL to which those citing publications referred, they “found them to be more disciplinary than might have been supposed. The Education publications that cited HPL were overwhelmingly more likely to cite content derived from the Education literature rather than from CogSci, whereas the CogSci publications that cited HPL were more likely to cite content derived from the CogSci literature”.

Sieweke (2014) used a similar approach as we applied to investigate the influence the French sociologist Pierre Bourdieu has in the field of management and organization research. The author did not only analyze the citations to Bourdieu’s publications over time, but also which contents from the publications were of greatest interest. The results show that “the three concepts of capital, habitus and field combined cover almost half of the citations” (Sieweke 2014, p. 537).

Methods

Datasets

In this study, we used citation context data from MA. Our current snapshot of the MA database contains bibliographic data of 212,209,775 documents, such as title, abstract, keywords, and citation relations to other documents inside the database. Our current snapshot contains 1,399,752,645 citation relations, 14.9% of them (n = 208,839,812) with citation context information. MA also provides an algorithmic assignment of documents to fields of study (FOS). The FOSs are ordered in a hierarchy of six levels (from level 0, broad research areas, to level 5, fine-grained disciplines). In our current snapshot, 178,899,665 documents are assigned to at least one of the 19 level 0 FOSs. Sinha et al. (2015), Hug et al. (2017), and Hug and Brandle (2017) provided more general information about MA.

We searched for the book titles to find all entries in MA which belong to the different editions of the three books by Kuhn and Popper. For Kuhn, the editions from 1962, 1970, and 1996 have two different entries, each, while the 50th anniversary edition from 2012 has only one entry. For Popper, we found one database entry each for the German and English editions of Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft and The logic of scientific discovery, respectively, and we found a single entry for Conjectures and refutations: the growth of scientific knowledge.

Certain terms—key concepts from Kuhn and Popper—have been searched in citances. We use the expression “citance” in this study although the citation context in MA frequently includes more than one sentence, or only a part of a sentence. The key concepts that we examine in this study from Kuhn’s book are the core concepts of his cyclical model of scientific change: paradigm, normal science, crisis, incommensurability, scientific revolutions, and structure. The key concepts from Popper’s books are all related to chapter titles. These are concepts that figured importantly in a number of debates in the philosophy of science: induction, falsification, demarcation, corroboration, and probability.

We searched the concepts in the citances in the following manner: In the first step, punctuation characters (“,”, “.”, “;”, and “:”) were removed from the citances and substituted with whitespaces, and all citances were converted to lower case characters. Second, a word frequency analysis was performed on the citances. Next, frequently occurring words, which describe common concepts employed by Kuhn (1962) or Popper (1934, 1959, 1962) were used to construct the search terms. For some concepts, the occurrence of the search terms was too low, and those concepts (e.g., “theory change” for Kuhn and “verifiability” for Popper) were dropped. We obtained a set of seven concepts for Kuhn and five concepts for Popper with an ample number of hits in the citances. The concepts and the corresponding search terms for Kuhn and Popper are shown in Table 1. In the case of the concepts “scientific revolution” and “structure”, the citances are excluded where the book title itself occurs. In the case of Popper, German search terms were employed as well.

Table 1 Concepts and corresponding search terms for Kuhn and Popper
Full size table

Unfortunately, the citation context is not available for all citing papers in MA. For Kuhn’s book, we found 38,474 citing papers, 4710 of them have citation context information available, and 4688 of the citing papers with citation context were assigned to at least one level 0 FOS—the highest level in the field categorization scheme used by MA. In total, we found 2426 papers citing Kuhn’s book which contained at least one of the search terms in Table 1. For Popper’s books, we found 14,829 citing papers, 2129 of them have citation context information available, and 2122 of the citing papers with citation context were assigned to at least one level 0 FOS. In total, we found 672 papers citing Popper’s books which contained at least one of the search terms in Table 1. Some of the citing papers referenced Popper and/or Kuhn multiple times and may be assigned to more than a single level 0 FOS.

In this study, we also employed an approach proposed by Small (2018) and Small et al. (2019) to measure uncertainty associated with concepts. Small et al. (2019) used the open access subset from PubMed Central and searched in the provided citation context data for hedging word strings (e.g., “may” and “could”). The authors note that “hedging does not assert that the paper is wrong, but only suggests that uncertainty surrounds some aspect of the ideas put forward” (Small et al. 2019, p. 1086). They computed hedging rates “separately for method and non-method papers, the latter being more frequently hedged. Rates of hedging are found to be higher for papers with fewer citances, suggesting that the certainty of scientific results is directly related to citation frequency” (Small et al. 2019). According to the results by Chen et al. (2018), fields with the highest rates of uncertainty are psychology, business, management, accounting, social sciences, economics, econometrics, and finance. Low rates can be found in mathematics, material science, and chemistry. Hyland (1996) distinguishes between reader-motivated (e.g., “believe”, “suggest”, and “analogy”) and content-motivated (e.g., “generally”, “almost”, “might”, and “probable”) hedging words. Uncertainty analysis is not without its problems. Small et al. (2019) provide a thorough discussion of problems encountered in undertaking uncertainty analyses.

Henry Small—one of the co-authors of Small et al. (2019)—provided us with his initial hedging words list for measuring uncertainty in his study. The list includes the following terms: “not clear”, “no clear”, “appears”, “possibility”, “seems”, “speculated”, “to some extent”, “impression”, “sometimes”, “perhaps”, “not known”, “seem”, “apparently”, “tends”, “not necessarily”, “preliminary”, “contingent”, “could”, “doubt”, “explore”, “feel”, “hope”, “hopeful”, “hopefully”, “likely”, “may”, “might”, “nevertheless”, “nonetheless”, “not known”, “opportunity”, “plausible”, “possible”, “possibly”, “potential”, “potentially”, “probable”, “probably”, “projected”, “promise”, “promising”, “questions”, “risky”, “speculative”, “suspect”, “uncertain”, “unclear”, “unknown”, “unsolved”, “whether”, and “yet to be determined”. We checked in the citances of our data set how frequently the terms occur. For the statistical analyses, we selected only those terms, which occur very frequently: “may”, “possible”, “could”, “questions”, “might”, “whether”, “potential”, “seems”, “perhaps”, “likely”, and “sometimes”. The resulting set was reduced when we eliminated terms for which we observed that they are frequently not used to express uncertainty in the citances. The final set is as follows: “like”, “may”, “could”, “questions”, “might”, “potential”, “seems”, “perhaps”, “likely”, and “sometimes”. The terms “possible” and “whether” were not used because many citances did not express uncertainty when these terms were used.

Besides measuring uncertainty, we also tried sentiment analyses of citation context data, but the results were not very promising. Thus, we decided not to include the results of the analysis in this manuscript. However, it would be interesting to explore whether certain concepts received a positive or negative connotation by the citing authors, similar to the uncertainty analysis.

Statistics

In this study, we basically count how often certain terms, i.e. concepts (e.g., “paradigm”), are mentioned in citances of citing publications. As more than one concept is frequently mentioned in single citances, it is necessary to consider in the statistical analyses the possibility that concepts are mentioned multiple times. The meta-data in MA allows us to analyze differences between fields and time periods in citing certain concepts by using the FOSs and publication years of the citing publications. The relationship of concepts and fields, or time periods, can be represented in a contingency table whereby the citations of concepts are dependent variables, and fields and time periods are independent variables.

We used the Stata command mrtab (Jann 2005) to analyze multiple mentions of concepts depending on FOS and publication year. On the one hand, we inspect differences in percentages of concept mentions between different FOSs and publication years as indications of effect size (Cumming and Calin-Jageman 2016). On the other hand, we calculate statistical significance tests in two-way tables: (1) We perform overall Chi squared tests to investigate the overall relationship between the cited concept and FOS, or publication year. In many analyses, however, the following assumptions for Chi squared tests on contingency tables are not fulfilled: “no more than 20% of the expected counts are less than 5 and all individual expected counts are 1 or greater” (Yates et al. 1999, p. 734). (2) We perform a series of separate Chi squared tests for each concept whereby the p values are adjusted to account for simultaneous testing.

The statistical significance tests can be used to check whether the results based on the samples (the available citances) can be used for drawing conclusions about the population (all citations of the books). The p values can be applied to make such inferences (Kühberger et al. 2014). In the interpretation of the results, however, it should be considered that the sample used in this study is based on the availability of citances in the database. Thus, the study is based on availability sampling, or so called “convenient sample” (which is different from a random sample). Since the representativeness of our sample is not clear, we used two variables (number of authors and number of countries) to test the representativeness of the convenient sample for the population. We selected these variables, since they were available for both the population and sample. The results are shown in Table 2 for three sets of papers citing Kuhn or Popper (citing papers with citances including hedging words, citing papers with citances, and citing papers) and all MA citing papers. With one exception, the results are very similar and indicate representativeness. The exception refers to the average number of authors of all MA citing papers (3.38). However, this is not the relevant population of our study (since we focus on citing papers of only three books). We have included it in the table only for purposes of comparison.

Table 2 Analyses of the representativeness of the sample for the population
Full size table

The analyzed papers in this study constituted a very small proportion of the papers citing Kuhn’s and Popper’s books. We identified 38,474 publications citing Kuhn; for only 4710 (i.e., 12.24%) we have citances, 4688 (i.e., 12.18%) are classified by FOS, and only 2426 (i.e., 6.3%) contain keywords we searched. That is, only 6.3% of citing papers were analyzed. The percentages for the publications citing Popper are 14.35%, 14.3% and 4.53% respectively. Furthermore, the keyword searching may fail to retrieve citances actually dealing with the concepts without using those specific keywords.

Results

We undertook a CCA for several landmark publications in philosophy of science: Kuhn (1962) and Popper (1934, 1959, 1962). First, we performed the CCA for various FOSs and time periods. Second, we analysed the (un-)certainty connected to the concepts. The results are presented in the following sub-sections.

First empirical part: citations of concepts in various fields of study (FOSs) and time periods

The results of the CCA for Kuhn (1962) are shown in Table 3. For each concept, the number of occurrences (citances) in each FOS are presented (absolute numbers and in percent). The line “Total” includes the total numbers of concept mentions in the citing papers’ citances (broken down by FOS). The line “Cases” refers to the total numbers of the citing papers’ citances (broken down by FOS). For many citing papers of the book, citation context information is available which does not contain any concept mentions considered here. The table reports column percentages based on all citances of the FOS, because we assume that the concept mentions are dependent on the FOS.

Table 3 Citation concept analysis of The structure of scientific revolutions by Kuhn. How frequently have concepts (the concepts are decreasingly sorted by the column “Total”) been cited in various fields of study (FOSs)?
Full size table

As the results in the table reveal, “paradigm” (with 40.44%) is the concept which is most frequently mentioned in citances of Kuhn (1962). Since this is the case in all FOSs, it seems that the book is mainly associated with the paradigm concept. With a large difference in numbers (around 30 percentage points), “paradigm” is followed by “normal science” with 5.66%.

In the notes of Table 3, the results of various Chi square tests are shown. The results for single concepts reveal whether the relationship between concept mentions and FOS is statistically significant (in cases where p < . 05). As the table shows, there are three statistically significant results, namely for the concepts “paradigm”, “normal science”, and “scientific revolution”. For example, “paradigm” has been more frequently mentioned in geography (with 58.33%) than in mathematics (with 31.05%). Since these results are based on convenient samples, it is not completely clear whether the results are valid in the population (of all citing papers). The result of the overall Chi squared test is not shown, since the requirements for running the test are not fulfilled.

Table 4 reveals the relationship between concepts and time periods for The structure of scientific revolutions (Kuhn 1962). These data enable us to determine if certain concepts have been increasingly mentioned over time. As the statistical significance tests show, the results for normal science are statistically significant. Between 2006 and 2010, the concept has been more frequently used (7.06%) than before 2000 (4.16%). The use of the other concepts in the citing literature scarcely changes over time.

Table 4 Citation concept analysis of The structure of scientific revolutions by Kuhn: How frequently have concepts (the concepts are decreasingly sorted by the column “Total”) been cited in various time periods (publication years)?
Full size table

Table 5 shows the results of the CCA for The logic of scientific discovery (in German: Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft) and Conjectures and refutations: the growth of scientific knowledge by Popper. “Falsification” is the concept which has been most frequently cited (24.23%). The results of the overall Chi squared test for the relationship between concept and FOS is not reported, since the requirements are not fulfilled. The partial results for “falsification” and “corroboration” are statistically significant. The results point out that the concepts have been cited at different rates in the various FOSs. For example, “falsification” has been more frequently cited in medicine (30.99%) than in sociology (18.58%). Again, since we used a convenient sample for the calculations, the validity of the findings for the population (all citances) may be questionable.

Table 5 Citation concept analysis of The logic of scientific discovery (in German: Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft) and Conjectures and refutations: the growth of scientific knowledge by Popper. How frequently have concepts (the concepts are decreasingly sorted by the column “Total”) been cited in various fields of study (FOSs)?
Full size table

With respect to Popper, Table 6 shows the relationship between the four concepts and various time periods. The Chi squared tests reveal a statistically significant result for the concept “corroboration”. However, we abstain from interpreting the result, since the case numbers are very low.

Table 6 Citation concept analysis of The logic of scientific discovery (in German: Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft) and Conjectures and refutations: the growth of scientific knowledge by Popper: How frequently have the concepts (the concepts are decreasingly sorted by the column “Total”) been cited in various time periods (publication years)?
Full size table

Table 7 shows the distribution of concept mentions of Kuhn and Popper in citances across FOSs (citations in citing papers with mentions of any of the concepts in the citation context data)—without considering the individual concepts. The data are from Table 3 and Table 5 (row: total N). We are interested in the meaning or significance of the books in various FOSs. The results of the Chi squared test indicates that the books are differently used (statistically significantly) in the FOSs. The table is decreasingly sorted by the absolute differences between the percentages for the books by each author in the table. The results reveal the largest difference for sociology: in general, Kuhn (1962) seems to be more important (17.26%) as a conceptual base than Popper (1934, 1959, 1962) (7.33%). In mathematics, we observe the opposite (3.95% versus 12.59%).

Table 7 Importance of The structure of scientific revolutions by Kuhn and The logic of scientific discovery (in German: Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft) and Conjectures and refutations: the growth of scientific knowledge by Popper in different fields of study (the FOS are decreasingly sorted by absolute differences between the percentages; the book with the greater importance is tagged with K = Kuhn or P = Popper)
Full size table

Abbott (2016) analyzed citations to Kuhn (1962) with respect to the scientific disciplines of the citing papers. He identified three main audiences of Kuhn: (i) philosophy, history, and philosophy of science, (ii) social sciences and psychology, and (iii) applied fields. Our results may confirm the high impact Kuhn’s concepts have had in sociology and psychology, although our results seem to show that Popper’s concepts are more important in psychology. The high impact of Kuhn (1962) that Abbott (2016) observed in history and philosophy might be less concept-specific, according to our results. Our results also point to a high impact of Kuhn’s concepts in applied fields, such as computer sciences, economics, and political sciences.

Second empirical part: uncertainty associated with concepts

Following an approach recently proposed by Small (2018) and Small et al. (2019), we measured uncertainty which might be associated with a particular concept. In the history of every scientific field, many concepts have been proposed. The literature dealing with these concepts provides information about the validity of the concepts: did the authors citing the concept confirm or question them? Using certain hedging words, we tried to catch uncertainty connected with the concepts investigated here in the citing literature. We expect that the conceptual approaches by Kuhn and Popper are differently associated with uncertainty. We also speculate that some concepts, but not others, are seen as certain knowledge. The results of the analyses considering various FOSs are shown in Table 8 for Kuhn, and Table 9 for Popper. In order to receive meaningful and interpretable results, only FOSs are considered with at least 300 concept citations in total. This meant for the CCA of Popper that FOS-specific analyses were not possible at all. The results in Table 8 reveal that around 15% of Kuhn’s concept mentions are connected to uncertainty; for Popper, the percentage is around 12% (see Table 9). Thus, it seems that Kuhn’s conceptual approach is slightly more frequently associated with uncertainty than Popper’s conceptual approach (in all FOSs).

Table 8 Counts (n and N) and percentage of citances—referring to the concepts in The structure of scientific revolutions by Kuhn—reflecting uncertainty in selected fields of study (FOSs; only FOSs are considered with at least 300 concept citations in total; the column “Total” refers to all—not only the selected—FOSs; the concepts are decreasingly sorted by the column “N”)
Full size table
Table 9 Counts (N) and percentage of citances—referring to the concepts in The logic of scientific discovery (in German: Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft) and Conjectures and refutations: the growth of scientific knowledge by Popper—reflecting uncertainty (all fields of study, FOSs; the concepts are decreasingly sorted by the column “N”)
Full size table

With around 21%, “crises” is the concept of Kuhn’s book which is more frequently associated with uncertainty than all other concepts; with around 11%, “normal science” is associated with the lowest rate of uncertainty. In the case of Popper, “corroboration” (14.81%) seems to be a more uncertain concept than “induction” (4%). For Kuhn, we also have FOS-specific results: in general, the most critical view on Kuhn seems to be in psychology (16.74%); the opposite is visible in economics (11.34%). “Paradigm” might be most frequently seen as uncertain in psychology (16.7%), and less frequently uncertain in economics (11.76%). Psychology seems to have the most critical view on “scientific revolution” (16.07%); for sociology, the percentage is much lower with 8.57%.

In another study, Atanassova, Rey, and Bertin (2018) reported more uncertainty and hedging word occurrences in bio-medical disciplines than in physics. The corresponding FOSs received too few concept citations in our study to draw a meaningful comparison.

In the context of the uncertainty analyses of the concepts by Kuhn and Popper, we also investigated developments over time. We are interested in whether the concepts are associated with an increasing or decreasing uncertainty. Since the numbers of cases for an analysis of the concepts introduced by Popper are too low, we refrain from the analysis on the basis of single concepts. The results for The structure of scientific revolutions by Kuhn in Table 10 show that the uncertainty seems to decrease over time (from 15.4% before 2000 to 14.31% in 2011 to 2018). This decreasing trend is also visible for “paradigm”. However, the reversed trend is also visible, e.g. for “crisis”.

Table 10 Counts (n and N) and percentage of citances—referring to the concepts in The structure of scientific revolutions by Kuhn—reflecting uncertainty (the concepts are decreasingly sorted by the column “Total”) in various time periods (aggregated publication years)
Full size table

Table 11 shows the results for the development over time for Popper. With the exception of the time period before 2000, the results also indicate, similar to Kuhn, a decreasing rate of uncertainty.

Table 11 Counts (N) and percentage of citances—referring to The logic of scientific discovery (in German: “Logik der Forschung: Zur Erkenntnistheorie der modernen Naturwissenschaft”) and Conjectures and refutations: the growth of scientific knowledge by Popper—reflecting uncertainty (increasingly sorted by the column “Time period”, aggregated publication years)
Full size table

Discussion

The use of citation counts in research evaluation is based on the norm that “not citing relevant prior work constituted a violation” (Zuckerman 2018). However, the focus on citation counts in quantitative research evaluation involves the danger of forgetting that a citation does not stand alone. It is embedded in the citing text which leads to different meanings of citations. The determination of different meanings is the objectives of citation context analyses. One goal of these studies is to develop methods allowing more detailed and informative citation analyses: which are, e.g., the papers in a specific field introducing important concepts which are frequently, intensively, and controversially discussed in a field? Quantitative (e.g., by using natural language processing techniques) and qualitative (e.g., classic close reading method) approaches can be performed for citation context analyses (Petrovich 2018).

In recent years, the full texts of papers are increasingly available electronically which opens up the possibility of quantitatively investigating citation contexts in more detail. According to Lamers et al. (2018) “of particular interest … is to determine if systematic analysis of citation context can help shed light on what role previous literature plays when cited in new publications, how new authors use past literature to further their own arguments, and whether we can disentangle disciplinary modes of knowledge accumulation from more general archetypes of contributions made to the scientific landscape” (p. 1115). Following the notion of a multi-dimensional perspective on citation impact proposed by Bu et al. (2019), we introduced in this study the CCA method for determining the citation impact that certain published concepts have had on the scientific community. As Petrovich (2018) notes: “citation context analysis seems particularly suitable to clarify the fine-grained structure of the knowledge accumulation process” (p. 1127). CCA can be used to determine the importance of scientific concepts published in highly-cited landmark publications.

It is worth reflecting on which of the concepts in Kuhn’s and Popper’s landmark books have left the largest impact, and on which fields of study. It is not surprising that Kuhn’s “paradigm” concept has had a significant impact. What is surprising is that it may have had such a disproportionately large impact compared to Kuhn’s other concepts. The paradigm concept accounts for around 40% of the citations to Kuhn’s work. And its impact seems resilient across all FOSs and over time. It is also not surprising that the second and third most popular concepts derived from Kuhn’s book are “normal science” and “scientific revolution”. Kuhn argued that both periods are an integral part of the development of a scientific field. Even though his concern was with the natural sciences, many scholars in the social sciences looked at their own fields of study in an effort to determine the extent to which Kuhn’s model of scientific development describes their own field (see Wray 2017).

Further investigation is needed in order to explain our finding that the concept of “normal science” may have had a slightly greater impact in mathematics (6.85%) than in physics (4.39%). One conjecture is that conceptual innovations in mathematics augment the existing knowledge, rather than replace what was regarded as secure knowledge before. For example, the discovery or development of non-Euclidean geometries did not render Euclidean geometry obsolete. In this way, all innovations in mathematics are continuous with a single normal scientific research tradition. But developments in physics are not like this. Newton’s physical theory led to the rejection of the contact physics associated with Galileo and Descartes. About two centuries later, Albert Einstein’s theory of general relativity led to a generalization of Newton’s gravitational theory. But Einstein’s innovations significantly changed our understanding of central concepts, such as “mass”, which is now regarded as convertible to energy, a significant change from the classical understanding of the concept. And the photon theory of light had a similar effect on the theories of light accepted by earlier generations of physicists. Normal scientific research traditions are interrupted in physics, as new theories replace older theories. It is worth remembering that Kuhn’s cyclical theory of scientific change was explicitly designed with the natural sciences in mind. He did not purport to be describing the dynamics of conceptual change in either the formal sciences, like mathematics, or the social sciences.

Similarly, with respect to Popper, it is not surprising that “falsification” seems to be the most used concept derived from his books. Falsification, after all, is the cornerstone of Popper’s critical rationalism. According to Popper (1934, 1959, 1962), the only way scientists can advance our scientific knowledge is by (i) attempting to falsify hypotheses, and (ii) rejecting those that are falsified. According to Popper (1934, 1959, 1962), falsifiability is also what distinguishes a science from a pseudo-science. Further investigation would be needed in order to explain why “corroboration” is used more in biology than in other disciplines.

Our finding points to a greater comparative impact for Kuhn’s book than Popper’s books in sociology, which is not surprising for two reasons. First, Kuhn (1962) develops a theory of science and scientific change that gives special attention to the changing social dynamics in scientific fields (see Wray 2011). The breakdown of the consensus in a scientific field that characterizes a pending scientific revolution or paradigm change is explicitly described in social terms. The field is described as being in crisis as the consensus breaks down. Even in Kuhn’s discussion of scientific revolutions, he explicitly draws a comparison between scientific revolutions and political revolutions. It is not surprising that this sort of analysis would appeal to sociologists. Popper, on the other hand, focuses narrowly on analyzing the logic of science, in keeping with the positivist tradition in which he both grew up and reacted against. Second, Popper had expressed disdain for the social sciences (see Wray 2017). Kuhn, on the other hand, avoided saying much about the social sciences.

Our finding that Popper’s books seem to have a greater impact than Kuhn’s book in mathematics is also not surprising. Kuhn, after all, was only concerned with the empirical natural sciences, not formal sciences, like mathematics. And Popper explicitly discusses probability theory in his book, thus making it relevant to at least some mathematicians, specifically those working in probability theory.

The fact that the level of uncertainty associated with the key concepts in Kuhn’s book decreases as we approach the present is not especially surprising. It could easily be a function of the fact that people now seem to have settled on how the terms should be interpreted or applied. Similar remarks explain the trends with respect to the uncertainty associated with the use of the key terms in Popper’s books. These books have existed long enough to have achieved canonical status in philosophy of science, and standardizing interpretations.

We have some concern about our method of measuring uncertainty and the level of uncertainty associated with two of Kuhn’s concepts. The term “crisis”, as Kuhn uses the term, is meant to convey a period of uncertainty in a scientific field, as anomalies persistently resist normalization. So the words we use to measure uncertainty may not be catching the uncertainty in the use of the term crisis, but rather the uncertainty associated with the period of crisis in a field. And a similar concern arises with respect to “normal science”. In periods of normal science, scientists take the conceptual scheme or theory for granted. The working assumption is that the theory reflects the structure of the world. Thus, some of the words we use to measure uncertainty may not be catching the uncertainty of the use of the term normal science, but rather the uncertainty that is associated with periods of normal science. The authors citing Kuhn may be using such words to describe normal science in their citances, rather than expressing a relatively low degree of uncertainty with Kuhn’s view of normal science. A more detailed study of these citances would need to be conducted to determine whether the uncertainty we measure reflects the relative uncertainty of the authors using the concepts.

As noted above, we recognize that our study has some limitations. The most severe limitation is probably the representativeness of the sample (see “Statistics” section): the MA database does not contain citation context information for all citing papers. For Popper, 14.4% of the citing papers have citation context information available. In the case of Kuhn, 12.2% of the citing papers have citation context information available. A less severe limitation is the selection of concepts and search terms for the concepts and the uncertainty detected. Here, also the wording and length of the citation context determines our ability to properly assign citation contexts to concepts and determine their uncertainty. Finally, a less worrying limitation is the fact that some publications are not assigned to FOSs in MA. As 99.7% of papers citing Popper’s books and 99.5% of papers citing Kuhn’s book were assigned to a level 0 FOS, this limitation seems negligible.

FOSs in MA are assigned algorithmically on the paper-basis. The quality and details about the algorithm of the FOS assignment are unclear (the same holds true for MA’s indexing criteria). Algorithmic FOS assignments may or may not be accurate. The accuracy of an algorithm based on direct citation relations has been questioned (Haunschild et al. 2018a, b). But a recent case study on computer science publications reported promising results regarding the MA FOSs (Scheidsteger et al. 2018). However, a large-scale comparison has yet to be conducted. Despite these limitations, we are confident that CCA is a very interesting and useful method. The books by Kuhn and Popper were used to show how the CCA method can be applied. If the missing citation context information is distributed randomly (without bias towards concepts, uncertainty words, publication years, and FOSs) our results should remain valid despite the mentioned limitations.