In Fabio Bonsignorio, John Hallam, Elena Messina & Angel P. Del Pobil (eds.), Metrics of sensory motor coordination and integration in robots and animals. Berlin: Springer. pp. 169-179 (2019)

Vincent C. Müller
Eindhoven University of Technology
While it is often said that robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion between a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment.
Keywords artificial intelligence  benchmark  Campbell's law  intelligence test  measure-target confusion  robotics  science
Categories No categories specified
(categorize this paper)
Buy the book Find it on
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

PhilArchive copy

 PhilArchive page | Other versions
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

Superintelligence: Paths, Dangers, Strategies.Nick Bostrom (ed.) - 2014 - Oxford University Press.
Critique of Pure Reason.I. Kant - 1787/1998 - Philosophy 59 (230):555-557.

View all 6 references / Add more references

Citations of this work BETA

No citations found.

Add more citations

Similar books and articles

AI Armageddon and the Three Laws of Robotics.Lee McCauley - 2007 - Ethics and Information Technology 9 (2):153-164.
Sustainability of Artificial Intelligence: Reconciling Human Rights with Legal Rights of Robots.Ammar Younas & Rehan Younas - forthcoming - In Zhyldyzbek Zhakshylykov & Aizhan Baibolot (eds.), Quality Time 18. Bishkek: International Alatoo University Kyrgyzstan. pp. 25-28.
Mental Robotics.Domenico Parisi - 2007 - In Antonio Chella & Riccardo Manzotti (eds.), Artificial Consciousness. Imprint Academic. pp. 191-211.
Of Robots and Believing.C. T. A. Schmidt - 2005 - Minds and Machines 15 (2):195-205.
Representation of Unattended Material in Memory.Yaakov Hoffman & Joseph Tzelgov - 2012 - Consciousness and Cognition 21 (3):1504-1508.
Institutional Robotics.Porfirio Silva & Pedro U. Lima - 2007 - In F. Almeida e Costa et al (ed.), Advances in Artificial Life. ECAL 2007. Springer Verlag.


Added to PP index

Total views
331 ( #30,783 of 2,498,320 )

Recent downloads (6 months)
22 ( #39,112 of 2,498,320 )

How can I increase my downloads?


My notes