Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 18 of 19 PEER REVIEW REPORT Review of: Ontologies relevant to behaviour change interventions: a method for their development [version 1; peer review: 1 not approved] Robert Kelly David Limbaugh Barry Smith Department of Philosophy, University at Buffalo 1. Introduction In "Ontologies Relevant to BCIs: A Method for their Development" Wright, et al. outline a step by step process for building ontologies of behaviour modification – what the authors call the Refined Ontology Developmental Method (RODM) – and demonstrate its use in the development of the Behaviour Change Intervention Ontology (BCIO). RODM is based on the principles of good ontology building used by the Open Biomedical Ontology (OBO) Foundry in addition to those outlined in (Arp, Smith, and Spear 2015). BCIO uses as its top-level ontology Basic Formal Ontology (BFO). The methods outlined in Wright, et al. are a valuable contribution to the field, especially the use of formal mechanisms for literature annotation and expert stakeholder review, and the BCIO will certainly play an important role in the extension of OBO Foundry ontologies into the behavioural domain. 1.1 Realist Methodology We shall concentrate here, however, on problems with the paper as it now stands, problems which arise primarily from a lack of emphasis on the realist methodology underlying BFO and the OBO Foundry. By 'realist methodology' we mean that terms in an ontology should refer to real entities and not (leaving aside ontologies of psychology) ideas or concepts in the minds of individuals (Smith and Ceusters 2010). For example, the term 'disease' should refer to whatever it is in the world that a disease is (in a specific patient) rather than referring to, for example, the idea of a disease in the mind of a clinician. The reason for this approach is that it promotes the interoperability and scalability of the ontologies developed – in much the way that a realist approach used in the sciences generally (which are not about the concepts in scientists' minds) promotes the development of scientific theories. The underlying assumption is that there are no contradictions in reality, and thus, if we carefully build ontologies in such a way that their terms refer to real entitles, then the results will be consistent, and thus interoperable, and the whole approach will be scalable to ever new areas or to ever more detailed levels of granularity. This approach has proved successful in the OBO Foundry (Kamdar, Tudorache, and Musen 2017). 1.1.1 Unification The realist approach is in the interest of unification. By 'unification' we mean, at least a shared vocabulary that Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 19 of 19 allows discussants to avoid merely verbal disputes, even if there is disagreement on what the world is like. That is, unification is approached when the building blocks used to assemble theories about reality are shared, even if experts assemble those blocks in different ways. As such, unification does not require absence of disagreement. It requires only that, when disagreement occurs, it is a disagreement over the nature of some phenomenon – of reality as it is – rather than over what some phenomenon is rightly called – for example, should we call it a 'mallard' or a 'duck'? An example of such a verbal dispute is provided by William James: The corpus of the dispute was a squirrel-a live squirrel supposed to be clinging to one side of a tree-trunk; while over against the tree's opposite side a human being was imagined to stand. This human witness tries to get sight of the squirrel by moving rapidly round the tree, but no matter how fast he goes, the squirrel moves as fast in the opposite direction, and always keeps the tree between himself and the man, so that never a glimpse of him is caught. The resultant metaphysical problem now is this: Does the man go round the squirrel or not? (James 1907, 34) The answer is, of course, it depends on what type of activity is properly referred to by 'go around', and were all of the senses of 'go around' distinguished and given appropriate labels and definitions, then all that would be left is a dispute about which sense of 'go around' is the right one to use in this instance: the discussants would have achieved unification. Unification is accomplished (at least) through having a shared vocabulary through which to express disagreement. For example, if one person claims that addiction is a disease while another claims that it is not, then to avoid merely verbal disputes it is important that 'addiction' and 'disease' mean the same thing to each person. Agreement of this sort is most readily achieved through the principle of low hanging fruit: start with vocabulary that reflects real entities unproblematically, and build up from there definitions of more complicated phenomena. 1.1.2 Scientifically Sound Now, while the authors never use the term 'realist' when they describe their methodologies, they do mention the need for ontological vocabularies to be scientifically sound. What the authors mean by 'scientifically sound' is unclear but seems to be: 1) terms are understandable by experts, and 2) terms are expressive enough to refer to all relevant entities in a field. While these conditions are important, they are not enough on their own to ensure that an ontology is realist, and thus not enough to build a good ontology. This is because, though experts in a field are generally extremely good at describing, to other experts in their field, entities relevant to their domain, they are not always good at describing them in a way that promotes the development of clear, organized, scalable, and interoperable vocabularies that can be understood also by experts in neighboring fields. The data-driven approach outlined by the authors to building and revising ontologies should, then, be paired with a process of carefully examining how terms are used by experts in order to understand the reality being referred to, and then adjusting and adding terms so that the ontology is both understandable to and usable by experts and such as to follow the structure of reality. 2 Specific Criticisms 2.1 Definitions of Key Terms Table 1 in Wright, et al. provides a glossary of key terms used in their article (as contrasted with terms in the BCIO). There are problems with some of the definitions of these terms, however, and in the spirit of complying with realist methods of ontology building and the goal of unification, it is useful if all relevant terms are treated as Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 20 of 19 if they are or will be included in an ontology. Artificial Intelligence: The theory and practice of building computer programs to perform tasks that a human would reasonably regard as requiring intelligence. According to the authors, 'Artificial Intelligence' is defined as a concept or piece of knowledge ("theory"), which is a continuant, and as a process ("practice"), which is an occurrent. Nothing can be both a continuant and an occurrent. By the authors' own lights, this is a bad definition because it traverses the two most fundamental categories of reality and thereby invites ambiguity when used in the article. However, to make matters worse, the authors then go on to use 'artificial intelligence' in what seems to be yet another sense: "Artificial Intelligence (AI) within the knowledge system will make predictions based on the evidence in response to users' queries about the most effective interventions in a wide variety of situations (e.g. type of behaviour, mode of delivery, population, setting)," which seems to suggest that AI is to be understood as an algorithm (p. 13). Interoperability: Ontology developers should collaborate with others wherever possible to re-use entities and limit duplication of work. Interoperability of ontologies sits within the OBO Foundry principle of Commitment to Collaboration. Unfortunately, this definition (or elucidation) does not correspond to the way 'interoperability' is predominantly used, according to which (in the simplest version): two systems are interoperable if data coming from each system can be used by the other system. What the authors provide, however, is not a definition of 'interoperability' but rather a prescriptive statement ("ontology developers should..."). Furthermore, it misunderstands the OBO Foundry principle of Commitment to Collaboration, which sites collaboration as having interoperability as one of its benefits, not as something that would be part of the definition of 'interoperability'. What the authors seem to be gesturing towards is a practice of making an ontology interoperable (in the standard sense) by using pre-existing terms and definitions where applicable. Thus, they are doing little more, here, than repeating the OBO Foundry's Commitment to Collaboration: An expectation that Foundry ontologies will collaborate with other Foundry ontologies, particularly in ensuring orthogonality of distinct ontologies, in re-using content from other ontologies in cross-product definitions where appropriate, and in establishing and evolving Foundry principles to advance the Foundry suite of ontologies to better serve the joint users (http://www.obofoundry.org/principles/fp-010collaboration.html). 2.2 Ontologies as Knowledge Graphs The authors sometimes describe ontologies as representing knowledge. This is, of course, true in the sense that ontologies do indeed provide proxies (for instance, in the form of definitions, and of is-a and other links in the ontology graph) for items of knowledge. If 'representing' is understood as meaning 'stand-in proxy for', then 'to represent knowledge' comes close to capturing the goal of realist ontology development. Unfortunately, however, there is another use of 'represent' and this yields a reading according to which terms in an ontology should be about or refer to knowledge, and this is antithetical to the realist methodology (just as 'ontologies represent concepts in people's minds' is antithetical to the realist methodology). More generally: terms like 'knowledge representation' should only ever be used with caution, because they run the risk of encouraging bad ontology development. Examples of Wright, et al.'s use of such terms are as follows (emphasis added): "Ontologies are knowledge structures ..." (abstract) or "ontologies encapsulate knowledge..." (p. 3). The same issue occurs in another paper developing the lower-level Mode of Delivery (MoD) Ontology that extends the BCIO (Marques, et al., "Delivering Behaviour Change Interventions: Development of a Mode of Delivery Ontology" [version 1]), and which shared a number of the same authors, where it is asserted that, "An Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 21 of 19 ontology is a more expressive structure for organizing knowledge," (p. 3, emphasis added) and that, "The research team developed relationships between ontology entities to formally capture the types of knowledge that are present in the ontology" (p. 6, emphasis added). 2.3 Lack of Conformance with Basic Formal Ontology Dispositions. Wright et al. define 'BCI scenario' as being both a disposition, which is a specifically dependent continuant in BFO, and a process, which is an occurrent. In line with the realist approach and principles of BFO and the OBO Foundry, no continuant is an occurrent and no occurrent is a continuant. a. 'BCI scenario': said to be the disposition (SDC) had by multiple entity types, but it is defined as a sub-type of planned process (occurrent). i. BCI scenario subclass_of planned process ii. BCI scenario plan realizes BCI scenario iii. BCI scenario report has_disposition BCI scenario Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 22 of 19 iv. BCI scenario plan realizes BCI scenario (see image above, under (ii)), and BCI scenario plan subclass_of information content entity b. 'outcome behaviour': said to be the disposition (SDC) had by some entity types, but it is defined as a sub-type of planned process (occurrent). i. BCI outcome estimate has_disposition outcome behaviour ii. BCI outcome estimate realizes BCI evaluation study, and BCI outcome estimate subclass_of information content entity (not a realizing entity) Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 23 of 19 iii. BCI outcome estimate realizes BCI evaluation study (image directly above), but BCI evaluation study realized_by BCI evaluation study plan (again, a non-realizing ICE, per image directly above) iv. outcome behaviour (said to be disposition) subclass_of process Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 24 of 19 c. 'BCI evaluation study': said to be the disposition (SDC) had by some entity types, but it is defined as a sub-type of planned process (occurrent). i. BCI evaluation study risk of bias or error and BCI evaluation report has_disposition BCI evaluation study Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 25 of 19 ii. BCI evaluation study subclass_of planned process d. 'BCI comparison evaluation study': said to be realized in (realizable entities are SDCs in BFO) some entity types, but it is defined as a sub-type of planned process (occurrent). i. BCI effect estimate (a non-realizing ICE) realizes BCI comparison evaluation study (a process) and BCI evaluation study (a process) ii. BCI comparison evaluation study subclass_of planned process, and also realized_in BCI evaluation study plan (another non-realizing ICE) iii. BCI comparison evaluation study has_study_investigator BCI study sample, making the study sample the same as the study investigator iv. BCI comparison evaluation study difference_between BCI scenario, which confuses the comparison of difference (process) and the difference itself (a quality) Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 26 of 19 B. Bearer_of Relation used incorrectly. Wright et al. use the bearer_of relation in their ontology in a way that is inconsistent with the realist principles of BFO and the OBO Foundry. They allow that some entities can be bearers of material entities (ICs) like object aggregates and systems. a. 'BCI context': said to be bearer_of BCI setting and BCI population, which are defined as object aggregate and human population, respectively. This entails that an object aggregate (e.g. a context) can bear a human population (an object aggregate). i. BCI context bearer_of BCI setting and BCI population ii. BCI context subclass_of object aggregate iii. BCI population subclass_of object aggregate iv. BCI setting subclass_of object aggregate b. 'BCI setting': said to be bearer_of BCI physical setting and BCI social setting, which are defined as environmental system and human population, respectively. This entails that an object aggregate (e.g. a setting) can bear a human population (object aggregate) or a system (material entity/IC) i. BCI setting bearer_of BCI physical setting and BCI social setting ii. BCI setting subclass_of object aggregate Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 27 of 19 iii. BCI physical setting subclass_of environmental system iv. BCI social setting subclass_of human population (object aggregate) C. Location of BCI Scenario: Wright et al. define BCI scenario as being located_in other occurrent entities in their ontology, some of which may be distinct, and even non-overlapping occurrent entities. Given the definition for the located_in relation, this is inconsistent since the "target" must be "entirely within" the location. a. Entirely Within: The located_in relation is defined as requiring the "target" to be "entirely within" the location. b. BCI scenario located_in BCI c. BCI scenario located_in BCI engagement d. BCI scenario located_in BCI mechanism of action e. BCI scenario located_in outcome behaviour (defined as a disposition and a process). This implies a process (BCI scenario) is located_in a disposition. Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 28 of 19 3. Conclusion In our view, Wright, et al.'s work developing the BCIO, as well as their RODM/SELAR3 method, constitutes a valuable and timely contribution to the field. As the authors point out, changing behaviors (whether at the individual, group, or organizational level) is required to improve public health and well-being, and to facilitate environmental sustainability (p. 3). Insofar as these things are valuable, so will be the need to study BCIs, their efficacy, how various factors like environment or technique influence their implementation or outcomes, and so forth. Hence, given the complexity of the phenomena surrounding BCIs, the volume of research and data on BCIs, and the heterogeneity of this data, we agree with Wright, et al. that development of the BCIO is an extremely important project to undertake. Importantly, the authors are committed to developing an ontology that is compliant with BFO and follows the principles of good ontology building used by the OBO Foundry. Indeed, Wright et al.'s RODM method outlined in the paper is strikingly analogous to the basic steps of building an ontology outlined in (Arp, Smith, & Spear 2015). For instance, Wright et al.'s method starts with defining the scope and identifying key entities, moves through a process of ensuring understandability and usability, and ends with specifying relations and making the ontology machine readable for dissemination, access, and maintenance. Arp and colleagues start with "demarcating the subject matter" and "gathering information," and then move to "regimenting the result to ensure" coherence, compatibly, and understandability, and end with "formalizing the regimented representational artifact" (Arp, Smith, and Spear 2015, p. 50). Moreover, in addition to the steps of development, Wright, et al. seem to subscribe to many of the core principles of good ontology design, like perspectivalism, fallibilism, re-use, and openendedness. Despite the importance of the authors' contribution, and without any intention of undermining that value, our review concentrated on the shortcomings of the ontology as it stands at present. In particular, we found that the BCIO is at present not fully compliant with BFO, nor does it yet fully adhere to the realist methodology underlying BFO and the OBO Foundry. It is this methodology that facilitates achieving unification in a discipline, and promotes the interoperability and scalability of the ontologies developed. This motivates our recommendation that the authors' data-driven approach be paired with a process of Wellcome Open Research 2020, 5:126 Last updated: 06 JUL 2020 Page 29 of 19 carefully examining how terms are used by experts in order to understand the reality being referred to, and then adjusting and adding terms so that the ontology is both understandable to and usable by experts and such as to follow the structure of reality. References Arp, Robert, Barry Smith, and Andrew D Spear. 2015. Building Ontologies With Basic Formal Ontology. Cambridge, Massachusetts: The MIT Press. International Organization for Standardization. 2020. ISO/IEC DIS 21838-2 Information Technology – Top-Level Ontologies (TLO) – Part 2: Basic Formal Ontology (BFO). Geneva. https://www.iso.org/standard/74572.html. James, William. 1907. Pragmatism A New Name for Some Old Ways of Thinking. Pragmatism. Auckland: The Floating Press. Kamdar, Maulik R, Tania Tudorache, and Mark A Musen. 2017. "A Systematic Analysis of Term Reuse and Term Overlap across Biomedical Ontologies." Semantic Web 8 (6): 853–871. https://doi.org/10.3233/sw-160238. Smith, Barry, and Werner Ceusters. 2010. "Ontological Realism." Applied Ontology 5 (3–4): 139–188. Prepared for Wellcome Open Research, July 6, 2020 https://wellcomeopenresearch.org/articles/5-126/v1#referee-response-