At the same time benchmarking is recognised as a major challenge in robotics – without benchmarks it is impossible to properly compare robot solutions – and several efforts have attempted to define and publish benchmarks (e.g., RawSeeds, BRICS, EURON SIG and IEEE TC on Experimental Methodology, Benchmarking, Challenges and Competitions).
This workshop is the third of a series of editions on Robot Competitions, following one at IROS 2012, and another at ERF 2013. This year it includes a panel of speakers – each leading competition challenge and/or benchmarking activities – to publically explore and debate the synergies between competitions, challenges and benchmarks. Ideally, competition scenarios, challenges and benchmarks should be co-created in order to both (a) utilise and test existing benchmarks in competitions and (b) use competitions to develop and extend benchmarks.
The detailed Workshop Programme is as follows:
08:30 – Welcome remarks by workshop co-chairs
08:35 – Gianluca Antonelli (EuRoC), The experience of the European Robotics Challenges
08.55 – Alan Winfield (euRathlon), Rules vs. benchmarks within euRathlon
09:10 – Matteo Matteucci and Pedro Lima (RoCKIn), RoCKIn approach to benchmarking
09:30 – Fabio Bonsignorio (TG on Experimental Methodology, Benchmarking, Challenges and Competitions), Replication of experiments and performance evaluation in robotics and cognition
09.50 – Anne Bajart (EC DG Connect), Benchmarking and performance evaluation in robotics: why this is important for Europe
10:00 – Panel discussion with the speakers and workshop audience
10:30 – Closing remarks (A. Winfield and P. Lima)