Measuring progress in robotics: Benchmarking and the ‘measure-target confusion’

In Fabio Bonsignorio, John Hallam, Elena Messina & Angel P. Del Pobil (eds.), Metrics of sensory motor coordination and integration in robots and animals. Berlin: Springer. pp. 169-179 (2019)
  Copy   BIBTEX


While it is often said that robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion between a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment.



External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Similar books and articles

AI armageddon and the three laws of robotics.Lee McCauley - 2007 - Ethics and Information Technology 9 (2):153-164.
Sustainability of Artificial Intelligence: Reconciling human rights with legal rights of robots.Ammar Younas & Rehan Younas - forthcoming - In Zhyldyzbek Zhakshylykov & Aizhan Baibolot (eds.), Quality Time 18. Bishkek: International Alatoo University Kyrgyzstan. pp. 25-28.
Mental robotics.Domenico Parisi - 2007 - In Antonio Chella & Riccardo Manzotti (eds.), Artificial Consciousness. Imprint Academic. pp. 191-211.
Of robots and believing.C. T. A. Schmidt - 2005 - Minds and Machines 15 (2):195-205.
Representation of unattended material in memory.Yaakov Hoffman & Joseph Tzelgov - 2012 - Consciousness and Cognition 21 (3):1504-1508.
Institutional Robotics.Porfirio Silva & Pedro U. Lima - 2007 - In F. Almeida e Costa et al (ed.), Advances in Artificial Life. ECAL 2007. Springer Verlag.


Added to PP

414 (#33,948)

6 months
35 (#60,688)

Historical graph of downloads
How can I increase my downloads?

Author's Profile

Vincent C. Müller
Universität Erlangen-Nürnberg

Citations of this work

No citations found.

Add more citations