Measuring the Effect of a Guideline-based Training on Ontology Design with a Competency Questions based Evaluation Approach

In M. Horbach (ed.), Informatik 2013. Informatik angepasst an Mensch, Organisation und Umwelt. pp. 1783-1795 (2013)
  Copy   BIBTEX

Abstract

OBJECTIVE: (a) To measure the effect of a guideline-based training on the performance of ontology developers compared with the performance after unspecific training by a competency question based evaluation; and (b) to provide empirical evidence for the applicability of competency questions in formal ontology evaluation in general. BACKGROUND: A close connection between ontology development and ontology evaluation as quality management procedure can been attained with the use of competency questions. Competency questions are often used as a semi-formal specification of requirements for an ontology under development. Hence they can also be used as evaluation instruments, in order to check how far an ontology fulfills these requirements. METHODS: A randomized controlled trial was conducted with two groups of 12 students each. The intervention consisted in a differential guideline-based training on ontology development vs. unspecific training. After a group-specific training focusing on three topics per group, performance of students was assessed with 12 exercises (2 exercises per topic), in which the students had to apply their skills. Different types of competency questions were elaborated for the analysis of the students' ontologies. We used the proportion of correct answers as a measure of ontology quality. RESULTS: On single topic level, the performance of ontology developers increased after guideline-based training for two out of the six topics: it increased from a proportion of 0.46 to 0.63 for the topic Process \& Participation and from 0.44 to 0.53 for the topic Collective Material Entity. In regression analysis, a positive correlation was shown between the performance of students on untrained topics and the performance after specific guideline-based training. Moreover, in multiple regression analysis an overall effect of specific training of 0.09 was calculated (p < 0.1). CONCLUSION: The results show an effect of a specific guideline-based training on the performance of ontology developers compared to the performance after unspecific training by an increase of about 10 \% on the rate of correct competency questions. In addition, this study has shown the general applicability of competency questions in a formal ontology evaluation scenario. However, the study also shows that the training of ontology developers and their performance evaluation is a tedious task. The resulting performance of ontology developers is more dependent on the a priori individual competencies than on the specific acquired skills after training.

Links

PhilArchive



    Upload a copy of this work     Papers currently archived: 90,616

External links

  • This entry has no external links. Add one.
Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

Definition und Evaluation einer Guideline zur Entwicklung von qualitativ guten Ontologien.M. Boeker, S. Schulz, D. Seddig-Raufie, D. Schober, J. Röhl, N. Grewe & L. Jansen - 2013 - GMDS 2013: 58. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biometrie Und Epidemiologie E.V. (GMDS). Lübeck 1.
Methodological Innovation in Practice-Based Design Doctorates.Joyce S. R. Yee - 2010 - Journal of Research Practice 6 (2):Article M15.
The doctor, his patient, and the computerized evidence‐based guideline.Toby Lipman - 2004 - Journal of Evaluation in Clinical Practice 10 (2):163-176.

Analytics

Added to PP
2014-02-21

Downloads
20 (#656,480)

6 months
1 (#1,042,085)

Historical graph of downloads
How can I increase my downloads?

Author's Profile

Ludger Jansen
PTH Brixen College

Citations of this work

No citations found.

Add more citations

References found in this work

No references found.

Add more references