Planner Metrics Should Satisfy Independence of Irrelevant Alternatives (position paper)

We argue that planner evaluation metrics should satisfy theindependence of irrelevant alternatives criterion, i.e., the de-cision whether planner A is ranked higher or lower than plan-ner B should be independent of planner C. We show that threemetrics used in classical planning competitions do not neces-sarily satisfy this criterion and highlight alternative metricsthat do so.