Journal of Biomedical Informatics 66 (2017) 129–135Contents lists available at ScienceDirect Journal of Biomedical Informatics journal homepage: www.elsevier .com/locate /y jb inDeveloping the Quantitative Histopathology Image Ontology (QHIO): A case study using the hot spot detection problemhttp://dx.doi.org/10.1016/j.jbi.2016.12.006 1532-0464/ 2016 Elsevier Inc. All rights reserved. ⇑ Corresponding author at: Department of Biomedical Informatics, OSU, 250 Lincoln Tower, 1800 Cannon Drive, Columbus, OH 43210, USA. E-mail address: metin.gurcan@osumc.edu (M.N. Gurcan). 1 HIMSS Dictionary of Healthcare Information Technology Terms, Acronyms and Organizations, 2nd Edition, 2010, Appendix B.Metin N. Gurcan a,⇑, John Tomaszewski b, James A. Overton c, Scott Doyle b, Alan Ruttenberg d, Barry Smith e aDepartment of Biomedical Informatics, The Ohio State University, Columbus, OH 43210, USA bDepartment of Pathology and Anatomical Sciences, University at Buffalo, Buffalo, NY 14214, USA cKnocean Inc., Toronto, Ontario M6P 2T3, Canada d School of Dental Medicine, University at Buffalo, Buffalo NY, 14214, USA eDepartment of Philosophy, University at Buffalo, Buffalo, NY 14260, USAa r t i c l e i n f o Article history: Received 24 August 2016 Revised 30 November 2016 Accepted 13 December 2016 Available online 18 December 2016 Keywords: Histopathology imaging Image analysis Hot spot Ontology Breast cancera b s t r a c t Interoperability across data sets is a key challenge for quantitative histopathological imaging. There is a need for an ontology that can support effective merging of pathological image data with associated clinical and demographic data. To foster organized, cross-disciplinary, information-driven collaborations in the pathological imaging field, we propose to develop an ontology to represent imaging data and methods used in pathological imaging and analysis, and call it Quantitative Histopathological Imaging Ontology – QHIO. We apply QHIO to breast cancer hot-spot detection with the goal of enhancing reliability of detection by promoting the sharing of data between image analysts.  2016 Elsevier Inc. All rights reserved.1. Introduction Interoperability across data sets is a key challenge for quantitative histopathological imaging (QHI). Interoperability describes the extent to which systems and devices can share data and interpret the data that is shared. The ideal is for multiple systems to be able to use and interpret each other's data in just the same way that they can use and interpret their own data. Limited interoperability between imaging data systems is a major obstacle to coherent multi-institutional collaboration, in digital pathology as in many other areas. Interoperability, according to the HIMSS Dictionary, may be seen on three different levels.1 At the foundational level interoperability describes the capability for simple data exchange from one information system to another, without any requirement for the receiving information technology system to be able to interpret the data that it receives. Interoperability at the structural level refers to the capability for data exchange in which the format and organization of the data is preserved unaltered. Here interoperabilityrelates to the syntax of the data exchanged. The highest level, that of semantic interoperability, is achieved according to the HIMSS Dictionary when data systems can take advantage ''of both the structuring of the data exchange and the codification of the data including vocabulary so that the receiving information technology systems can interpret the data." Many strategies to achieve semantic interoperability nowadays require the use of controlled vocabularies which provide single annotations or tags to be used to address the problems which arise when multiple coding systems use different codes describe the same entities in reality. ''Ontologies" improve on controlled vocabularies by using links and logical definitions to connect terms in a rich network of well-defined relationships. With the advance of pathological imaging technology and of associated software for the processing of pathological images, the need arises for an ontology which can support effective merging of pathological image data with associated clinical and demographic data that have already been described using existing controlled vocabularies such as SNOMED-CT, the NCI Thesaurus, or the ontologies such as the Cell Ontology constituting the OBO Foundry [1]. To this end we are constructing a Quantitative Histopathological Image Ontology (QHIO) incorporating terms representing the different types and subtypes of pathological images, imaging processes and techniques, and computational algorithms. In addition the ontology will incorporate formal definitions of these terms and specify 130 M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–135formally the relations that hold between entities of the corresponding types. Because QHIO will itself follow the principles of the OBO Foundry, the data resulting from the use of QHIO terms in annotations will be in a form that allows integration with other commonly used ontologies in the biomedical domain. The result will allow us to leverage the opportunities brought by new imaging platforms and algorithms to create an environment in which the clinical imaging data, and clinician and algorithmically created annotations deriving from different communities of clinicians and scientists, can be combined and analyzed as a single whole.2. The problem of reproducibility of image analysis A further urgent factor in contemporary research is the issue of reproducibility of clinical and scientific findings. The reproducibility and validation of large-scale, cross-institutional imaging research is limited by the fact that there is lacking any common structured framework for describing images and the results of their analysis. Here, too, we believe, ontologies can play a role by providing controlled vocabularies which can be used to describe in standardized ways the steps taken to achieve particular results [2]. Currently, pathology image data is collected in local ''silos" using in-house protocols, and is processed using proprietary algorithms developed in isolation. Typically the software itself may be unavailable to downstream image consumers, and even when it is available, there is rarely information about the sets of parameters necessary to run the software. Even the type of outputs of these algorithms, for example, a new potential prognostic feature, is not open to discovery by third party software. For these and a series of related reasons digital histopathology is difficult to reproduce and validate. While reproducibility and validation are important goals in their own right, even more value can be gained by combining and building upon image data and software methods across institutions.3. Quantitative Histopathological Imaging Ontology (QHIO) To foster organized, cross-disciplinary, information-driven collaborations in the pathological imaging field, we propose to develop an ontology to represent imaging data and methods used in pathological imaging and analysis with an initial focus on enabling effective communication and collaboration between developers of algorithms for analyzing histopathology images. Our ontology is modeled after the Gene Ontology (GO) [3] and follows the principles of the Open Biomedical Ontologies (OBO) Foundry [4], principles now used by some dozens of ontology-driven research efforts in clinical and translational science. An essential aspect of our work is the reuse of existing ontology terms wherever possible, thereby increasing the potential for interoperability as well as reducing the effort involved in defining new terms [5]. Hereafter when we refer to QHIO we mean the aggregate of terms we define, together with those terms that have been reused (imported) from other ontologies [5]. We intend to continue to extend and use QHIO to promote longterm interoperability of data across pathology imaging. We will cultivate wide community dissemination through development of model collections that include sample image data, example software that can compute useful information about them as well as the annotated output of this software. Both the model collections and the ontology will be released into the public domain, again following the model of the GO [4]. The workflow in histopathology-based clinical testing extends from biopsy collection, to slide production and analysis, to the assignment of a diagnostic category, to the creation of a predictivestatement for an individual patient based on that diagnostic category. All of these activities combine to support the choice of a treatment plan by a clinician for a particular patient. At the same time these activities cover a variety of different domains and thus they raise multiple questions that will impact interoperability. Some of these questions include: How are demographic, anatomical and relevant health information to be gathered and communicated? How are annotations for histology image data to be created and what will they identify? What information will be needed to enable productive communication between image analysis researchers and clinicians? Answers to these questions will influence the success and quality of interoperability. Our long-term goal is to demonstrate that an ontology can enable interoperability in the pathology image domain. To achieve this goal, we are in the process of curating an annotated dataset and developing parallel sets of algorithms for quantifying breast cancer histopathology. We will record the results of running different combinations of these algorithms on the same dataset as a test of their utility, while at the same time developing and testing QHIO itself. QHIO's overarching goal is to foster interoperability of computationally derived image data. We need to emphasize that this study is not designed so much to help routine clinical diagnosis; rather it is designed to create an enriched data resource in which various kinds of clinical and translational software approaches can be tested out. It can also be used to explore the role image data may play in clinical decision support systems and to test hypotheses regarding correlations between image features and patient outcomes, or between image features and other features catalogued by OBO Foundry ontologies.4. Motivating examples We present two general activities (i.e. validation and combining multiple analyses to make diagnoses), and give a more detailed use case – the hot spot detection problem – that motivates our effort.4.1. Activity 1: Validation The issue of interoperability is a major concern in validation. For example, two groups working on feature extraction may be investigating different but overlapping features, but currently have no way to develop a comparison tool to use their extracted features on the same dataset and thus produce quantitative results in a manner that can be easily communicated. This is important because such comparison is not only important for the qualitative validation of subject matter experts' prognoses but also for timely identification of divergence of results flowing from use of independently developed algorithms targeting the same object.4.2. Activity 2: Combining multiple analyses to make diagnoses Another scenario where QHIO might be useful is in the making of diagnoses on the basis of data derived from multiple analyses. Several different algorithms can be run on the same images to analyse different aspects (e.g. the amount of stroma and mitotic count). A diagnostic program needs to check whether the available data have the specific features it needs: relative amounts of cells by type, quantity of mitotic cells, and potentially a subgroup of those within certain regions. It is important that we can establish effectively that these features are among those found in several different analyses while the algorithm determines the case, retrieves the relevant information, and computes the diagnostic in a robust manner. M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–135 1314.3. Use case: The hot spot detection problem Ki67, a nuclear marker expressed in all phases of the cell cycle except G0 [6], has been widely used in pathology to assess proliferation within multiple neoplasms. ''Hot Spots", are areas with prevalent Ki67 staining with the highest number of positively staining nuclei within the invasive component. In the literature, there is no uniform approach to scoring hot spots. Many studies have specifically targeted hot spots in determining the Ki67 index while others have done an overall assessment of Ki67 with the incorporation of the hot spots in the overall index. Clearly, studies addressing outcome with detailed hot spot scoring are warranted. However in order to do better we need to develop an automatic hot spot detection method that can minimize the intraand interpathologist variability in identification of hot spots. In previous work, we developed a hot spot tool [7] for breast cancer as Ki67 has shown promise as a prognostic marker and predictor of responsiveness to chemotherapy or endocrine therapy. The method first segments Ki67 positive pixels using a previously-developed Visually Meaningful Segmentation (VMS) method [8]. VMS generates an image-dependent filter, which in turn generates a density map from the segmented image. The smoothness of the density image simplifies the detection of local maxima, which directly correspond to the hot spots in the image. The method was tested on 23 different regions of interest extracted from 10 breast cancer Ki67 slide images. To determine intra-reader variability, each image was annotated twice for hot spots by a board-certified pathologist with a two-week interval between the two readings. A computer-generated hot spot was considered true-positive if it agreed with either of the two annotation sets provided by the pathologist. While intra-reader variability was 57%, our method correctly detected hot spots with 81% precision. In order to run this tool at multiple institutions, some interoperability issues need to be overcome. For instance, we need a way to identify the stain type as Ki67. Additionally, several terms need to be defined and communicated, for example, positive and negative stain, hot spot boundary, image magnification, etc., both to the algorithm as well as the pathologist who will use this system.5. Methods In our paper ''Biomedical imaging ontologies: A survey and proposal for future work" [9] we surveyed the state of the art as concerns the development and application of controlled vocabularies and ontologies for digital pathology [10,11]. We also laid out a plan for building QHIO and for using QHIO in promoting the sharing of pathology image data and associated algorithm and algorithm output information. We have begun to execute this plan by 1. creating a prototype of QHIO and 2. applying it to breast cancer hot-spot detection with the goal of enhancing reliability of detection by promoting the sharing of data between image analysts. Our goal here is to demonstrate that QHIO can be used for pathology data sharing, and that it can serve as a starting point for further development toward realizing our longer term goals of advancing interoperability of histopathological imaging systems and reproducibility of histopathological imaging assays. The current version of QHIO is available at https://github.com/ontodev/ QHIO. A turning point in the development of ontologies and their extensive use in biology and biomedicine was the building and application of the Gene Ontology (GO) [3]. That work showed the benefits of tagging sequence data obtained from both humansand multiple model organism species with a single set of species-neutral terms. The success of the GO created a situation in which many biomedical subdisciplines saw a need to develop ontologies of their own, often in uncoordinated fashion with resultant tendencies to forking and redundancy. To counteract these tendencies a group of researchers developing ontologies centered on the GO established, in 2004, the Open Biomedical Ontology (OBO) Foundry initiative, promulgating a set of principles for ontology development which have been tested in practice and refined in light of the lessons learned by the many groups who have since sought to apply them in their work. It is these principles which we have used also to guide our work on QHIO. Chief among them is a commitment by the developers of each ontology to ensure interoperability with its neighbouring ontologies.5.1. The Ontology for Biomedical Investigations One neighbour to QHIO is the Ontology for Biomedical Investigations (OBI), an OBO Foundry ontology with over 2500 terms for describing biomedical investigations, including core terms such as investigation, assay, planning, protocol, specimen, and conclusion based on data. Currently, OBI contains a small number of terms for medical imaging and pathology, including terms such as imaging assay, staining, feature extraction, and pathologist role. We are developing QHIO as an extension of OBI, with the immediate advantage of interoperability between QHIO and the larger OBO ecosystem. QHIO will be subject to the same processes of ongoing review and maintenance and be able to draw on the deep expertise that the OBI community has developed over several years. OBI's hierarchy of terms can be used to represent investigations either in great detail or in broad strokes, as appropriate. For example, OBI can be used to annotate a given body of experiment records along dimensions such as: funding agency, antibodies, staining methods, statistical algorithms used, and so forth. In this way OBI enables powerful querying across experimental data, made still more powerful through cross-linkage to other OBO Foundry ontologies such as the Gene Ontology, the Cell Ontology (CL) [12], the Foundational Model of Anatomy (FMA) [13], the Human Disease Ontology [14], Chemical Entities of Biological Interest (ChEBI) ontology [15], as well as to external ontology resources including – most importantly for our purposes – the National Cancer Institute Thesaurus (NCIT). This enables other researchers to find experimental data that have been annotated with OBI because they know which terms to look for when searching. It also enables other researchers to understand how the data was acquired, since OBI enables provenance information pertaining not only to the persons, organizations, places and times of generation of the data, but also the experimental and data processing methods used. A number of initiatives are under way to map these and other OBO Foundry ontologies to major clinical vocabularies (especially SNOMED CT and ICD) in order to allow them to be used for querying and analytical purposes in tandem with Electronic Health Record and other clinical data [16]. We will take this work further in the specific field of breast cancer research, taking account of the set of common data elements (CDEs) recommended by the NCI's Early Detection Research Network (EDRN) for the treatment of breast cancer biomarker data. We will explore applications of our work to the creation of CDEs for cancer imaging biomarkers, drawing also on the preliminary work performed within the framework of the Quantitative Imaging Biomarker Ontology (QIBO) project [17]. Fig. 1. From specimen to hot spot detection. Overview of types and relationships in QHIO. Types are taken from OBI. More specific types from QHIO noted after colon. Fig. 2. Sample workflow represented through QHIO and OBI terms. (PCA = Principal Component Analysis). 132 M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–1355.2. What will QHIO achieve? Currently whole slide pathology images are annotated using vendor-supplied, freely available software programs. The resultant annotations in these programs typically employ natural language phrases, sometimes using machine-readable formats such as XML. They are limited to small controlled vocabularies of terms for colors and basic geometrical shapes (e.g. lines, circles, freeform figures), and use unstructured text fields to describe features of interest (e.g. 'necrotic region', 'necrosis', etc.). Annotations created by members of given communities using given vendor software are in almost every case incompatible with those created by other communities using other software. This locks valuable data into information silos. The challenges of sharing algorithms and image features are even greater than those of sharing image annotations because no standards currently exist to describe algorithms and image features. To rectify this problem QHIO will include standard terms not merely for algorithm and feature types and attributes but alsofor all other entities involved in each stage of the pathological imaging and analysis workflow (Fig. 1). These will include: 1. Input: type (e.g. image), preparation processes such as staining, slide characteristics, magnification, resolution, the meaning and typical range of parameters and other annotations. 2. Parameters: size of the filter window sizes, the number of iterations, etc. 3. Output: type (e.g. image, measurement), the meaning and typical range of parameters, statistical details (e.g. accuracy, false positive rate), statistical evaluation methodology (e.g. ROC) and the methods/software required. 4. Execution: operating system (e.g. Windows 7), software environment (e.g. Matlab), required resources (e.g. RAM, storage), expected time to run the algorithm. In addition there is a need to improve the existing annotation scheme by using QHIO in tandem with standard linked data formats that generalize current approaches based on XML, by Table 1 Hot spot detection algorithm related terms, existing OBO superclass and URI, and upper level term. Information entity is from the IAO, disorder from OGMS, and rest of the terms are from BFO. Term needed Mid-level parent Identifier High-level parent 1 Color space transformation Data transformation OBI_0200000 Process 2 Image segmentation Biological feature identification OBI_0000015 Process 3 Object classification Class discovery data transformation OBI_0200175 Process 4 Clustering Class discovery data transformation OBI_0200175 Process 5 Pixel Unit UO_0000000 Information entity 6 Image markup Symbol IAO_0000028 Information entity 7 Image annotation Textual entity IAO_0000300 Information entity 8 Histological slide scanning Image creation OBI_0001007 Process 9 Histological slide scanner Image creation device OBI_0000398 Material entity 10 Image region Part_of some image IAO_0000101 Information entity 11 Histological mounting Histological sample preparation OBI_0000341 Process 12 Ischemia Ischemia DOID_326 Disorder 13 Tissue dehydration Material processing OBI_0000094 Process 14 Histological sectioning Histological sample preparation OBI_0000341 Process 15 Optical magnification ratio Device setting OBI_0000654 Quality 16 Microns per pixel Unit label UO_0000000 Information entity 17 Image annotation creation Documenting IAO_0000572 Process 18 Histological slide Microscope slide OBI_0400170 Material entity 19 Ki67 stain Cytological stain role OBI_0000026 Material entity 20 Ki67 Antigen Ki-67 PR_000010425 Material entity 21 Hot spot Cellular feature identification OBI_0000219 Information entity 22 Histological section Tissue specimen OBI_0001479 Material entity 23 Mitosis count Substance unit UO_0000006 Information entity 24 Ki67 percentage Concentration unit UO_0000051 Information entity Fig. 3. An example representation in the QHIO prototype generated using Protégé. M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–135 133 134 M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–135applying QHIO and associated software to make image and algorithm annotations cleanly interoperable between sites (Fig. 2). For this study, we have identified hot spot detection algorithmrelated terms (Table 1). These terms as well as ontological terms necessary to represent images were included in the QHIO prototype (Fig. 3).6. Discussion and conclusion Experience shows that – even with the use of traditional Delphi techniques [18] – current image annotation formats and current ways of creating annotations (in many cases on the basis of free text or locally developed codes) leave much of the data accessible only at the site where it was created. Such data is both difficult to interpret and understand outside that context, and is in practice thus undiscoverable by external researchers. Our future work will be about bridging that gap by building an ontology and supporting software for better data sharing. The potential value of QHIO is its ability to aggregate data from multiple sources that can then be subject to analysis for research purposes. Establishing which images are compatible with a chosen algorithm and which might be pooled in an aggregate analysis certainly could benefit researchers, and the provision of corresponding information in the QHIO would be very valuable for this purpose, whether embedded in the image metadata or in the database from which they are derived. Using a subset of the common dataset, ontology development experts need to continue the development of QHIO. QHIO must be applied to convert the annotated image data into a set of interoperable, standardized data resources, supported by tools that allow for rich search and analysis, and automated matching of algorithms to appropriate input data. Ontologies serve as crucial aids to human communication, and are supported by a range of technologies that aid machine communication, search, and analysis [19]. We will use linked data standards developed by the World Wide Web Consortium (W3C) and associated technologies to build, maintain and use QHIO, and to process, store, and query annotations on images and algorithms. In particular the W3C standard Web Ontology Language (OWL) [20] will serve as the foundational technology for QHIO, as it is for all OBO ontologies. OWL brings both the ability to define and organize the classes in the ontology and the ability to instantiate those classes in the case of particular workflows or studies. OWL builds on the Resource Description Framework (RDF), which can be queried using the SPARQL Query Language. These and other software resources used in our project are based on open source tools and libraries. We have already demonstrated the use of OWL in QHIO, and used it to model the application of a hot-spot detection algorithm. QHIO will also build upon existing work done by the Open Microscopy Environment project (http://www.openmicroscopy. org/site/support/ome-model/). For example, OME already deals very effectively with the details of various file formats. QHIO and OME are complementary efforts. One future research direction is to establish tight integration between OME and QHIO, although this may require changes to both OME and QHIO. Additionally, we need to supply resources for keeping track of the image artifact itself, of how it changes as different sorts of annotations are added. We will use QHIO and linked data tools to develop an automated workflow for converting existing image annotations in our common dataset to linked data annotations using QHIO terms and terms from associated ontologies and structured resources referencing clinical data. The converted data will be loaded into an RDF database where it can be shared and queried using SPARQL. It is at this point the benefits of using OBO Foundry ontologies become increasingly clear: Every ontology term belongs to a hierarchy of more-and-less general terms, allowing for queries at many levels of specificity.  Ontology terms are defined through logical axioms that provide further links across the network of data, such as parthood, aboutness, adjacency, inclusion and other relations.  The logic of OWL allows for automated reasoning software to infer further links in the network from existing links and logical axioms, facilitating error checking and discovery of new knowledge.  Pathology imaging data can be easily integrated with data from other medical and scientific domains. The technology that is involved is now widely dispersed through the biomedical research community through the sustained support not merely of the W3C [20] but also of the National Cancer Institute, the International Health Terminology Standards Development Organization (IHTSDO, now responsible for SNOMED – CT) and by other bodies.Funding The project described was supported in part by Awards Number U24CA199374 (PIs: Gurcan, Madabushi, Martel) from the National Cancer Institute and UL1TR001412 (co-PI: Smith) from National Center for Advancing Translational Sciences (NCATS). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute, National Center for Advancing Translational Sciences (NCATS), or the National Institutes of Health.Conflict of interest statement The authors declare that they have no conflict of interest. The NIH funding sources have been specified in the acknowledgement section. References [1] A.D. Diehl, T.F. Meehan, Y.M. Bradford, M.H. Brush, W.M. Dahdul, D.S. Dougall, et al., The Cell Ontology 2016: enhanced content, modularization, and ontology interoperability, J. Biomed. Seman. 7 (2016) 44. [2] R.R. Brinkman, M. Courtot, D. Derom, J. Fostel, Y. He, P.W. Lord, et al., Modeling biomedical experimental processes with OBI, J. Biomed. Seman. 1 (2010) S7. [3] M. Ashburner, C.A. Ball, J.A. Blake, D. Botstein, H. Butler, J.M. Cherry, et al., Gene Ontology: tool for the unification of biology, Nat. Genet. 25 (2000) 25–29. [4] B. Smith, M. Ashburner, C. Rosse, J. Bard, W. Bug, W. Ceusters, et al., The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration, Nat. Biotechnol. 25 (2007) 1251–1255. [5] M. Courtot, F. Gibson, A.L. Lister, J. Malone, D. Schober, R.R. Brinkman, et al., MIREOT: The minimum information to reference an external ontology term, Appl. Ontol. 6 (2011) 23–33. [6] J. Gerdes, H. Lemke, H. Baisch, H.-H. Wacker, U. Schwab, H. Stein, Cell cycle analysis of a cell proliferation-associated human nuclear antigen defined by the monoclonal antibody Ki-67, J. Immunol. 133 (1984) 1710–1715. [7] M.K.K. Niazi, E. Downs-Kelly, M.N. Gurcan, Hot spot detection for breast cancer in Ki-67 stained slides: image dependent filtering approach, in SPIE Medical Imaging, 2014, pp. 904106–904106-8. [8] M. Niazi, G. Beamer, M.N. Gurcan, Detecting and characterizing cellular responses to Mycobacterium tuberculosis from histology slides, Cytometry Part A 85 (2014) 151–161. [9] B. Smith, S. Arabandi, M. Brochhausen, M. Calhoun, P. Ciccarese, S. Doyle, et al., Biomedical imaging ontologies: a survey and proposal for future work, J. Pathol. Inform. 6 (2015). [10] S. Adebayo, K. McLeod, I. Tudose, D. Osumi-Sutherland, T. Burdett, R. Baldock, et al., PhenoImageShare: an image annotation and query infrastructure, J. Biomed. Seman. 7 (2016) 35. [11] A. Galton, G. Landini, D. Randell, S. Fouad, Ontological Levels in Histological Imaging, in: Formal Ontology in Information Systems: Proceedings of the 9th International Conference (FOIS 2016), 2016, p. 271. [12] T.F. Meehan, A.M. Masci, A. Abdulla, L.G. Cowell, J.A. Blake, C.J. Mungall, et al., Logical development of the cell ontology, BMC Bioinform. 12 (2011) 6. [13] C. Rosse, J.L. Mejino Jr, A reference ontology for biomedical informatics: the Foundational Model of Anatomy, J. Biomed. Inform. 36 (2003) 478–500. M.N. Gurcan et al. / Journal of Biomedical Informatics 66 (2017) 129–135 135[14] L.M. Schriml, C. Arze, S. Nadendla, Y.-W.W. Chang, M. Mazaitis, V. Felix, et al., Disease Ontology: a backbone for disease semantic integration, Nucl. Acids Res. 40 (2012) D940–D946. [15] J. Hastings, P. de Matos, A. Dekker, M. Ennis, B. Harsha, N. Kale, et al., The ChEBI reference database and ontology for biologically relevant chemistry: enhancements for 2013, Nucl. Acids Res. 41 (2013) D456–D463. [16] P.N. Robinson, Computational phenotype analysis in human medicine, Phenomics (2014) 8. [17] A.J. Buckler, M. Ouellette, J. Danagoulian, G. Wernsing, T.T. Liu, E. Savig, et al., Quantitative Imaging Biomarker Ontology (QIBO) for knowledgerepresentation of biomedical imaging biomarkers, J. Digit. Imaging 26 (2013) 630–641. [18] N. Bowles, The Delphi technique, Nurs. Stand. 13 (1999) pp. 32–36 (1999/07/ 28). [19] P.L. Whetzel, N.F. Noy, N.H. Shah, P.R. Alexander, C. Nyulas, T. Tudorache, et al., BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications, Nucleic Acids Res. 39 (2011) W541–W545. [20] (2012). OWL 2 Web Ontology Language Document Overview. Available: <http://www.w3.org/TR/owl2-overview/>.