Measuring progress in robotics: Benchmarking and the ‘measure-target confusion’

In Fabio Bonsignorio, John Hallam, Elena Messina & Angel P. Del Pobil (eds.), Metrics of sensory motor coordination and integration in robots and animals. Berlin: Springer (forthcoming)
Authors
Vincent C. Müller
University of Leeds
Abstract
While it is often said that robotics should aspire to reproduc- ible and measurable results that allow benchmarking, I argue that a fo- cus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion be- tween a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment.
Keywords artificial intelligence  benchmark  Campbell's law  intelligence test  measure-target confusion  robotics  science
Categories (categorize this paper)
Options
Edit this record
Mark as duplicate
Export citation
Find it on Scholar
Request removal from index
Revision history

Download options

Our Archive
External links

Setup an account with your affiliations in order to access resources via your University's proxy server
Configure custom proxy (use this if your affiliation does not provide a proxy)
Through your library

References found in this work BETA

No references found.

Add more references

Citations of this work BETA

No citations found.

Add more citations

Similar books and articles

AI Armageddon and the Three Laws of Robotics.Lee McCauley - 2007 - Ethics and Information Technology 9 (2):153-164.
Sustainability of Artificial Intelligence: Reconciling Human Rights with Legal Rights of Robots.Ammar Younas & Rehan Younas - forthcoming - In Zhyldyzbek Zhakshylykov & Aizhan Baibolot (eds.), Quality Time 18. Bishkek: International Alatoo University Kyrgyzstan. pp. 25-28.
Mental Robotics.Domenico Parisi - 2007 - In Antonio Chella & Riccardo Manzotti (eds.), Artificial Consciousness. Imprint Academic. pp. 191-211.
Of Robots and Believing.C. T. A. Schmidt - 2005 - Minds and Machines 15 (2):195-205.
Representation of Unattended Material in Memory.Yaakov Hoffman & Joseph Tzelgov - 2012 - Consciousness and Cognition 21 (3):1504-1508.
Institutional Robotics.Porfirio Silva & Pedro U. Lima - 2007 - In F. Almeida e Costa et al (ed.), Advances in Artificial Life. ECAL 2007. Springer Verlag.

Analytics

Added to PP index
2018-06-09

Total downloads
35 ( #171,747 of 2,260,094 )

Recent downloads (6 months)
35 ( #10,844 of 2,260,094 )

How can I increase my downloads?

Monthly downloads

My notes

Sign in to use this feature