Action Completion

July 2016

Farnoosh Heidarivincheh, Majid Mirmehdi, Dima Damen

Action Completion - Overview

Robust motion representations for action recognition have achieved remarkable performance in both controlled and in-the-wild scenarios.

Such representations are primarily assessed for their ability to label a sequence according to some predefined action classes (e.g. walk, wave, open). Although increasingly accurate, these classifiers are likely to label a sequence, even if the action has not been fully completed, because the motion observed is similar enough to the training set. Consider the case where one attempts to drink but realises the beverage is too hot. A drinking-vs-all classifier is likely to recognise this action as drinking regardless.

We introduce the term action completion as a step beyond the task of action recognition. It aims to recognise whether the action's goal has been successfully achieved. The notion of completion differs per action and could be infeasible to verify using a visual sensor, however, for many actions, an observer would be able to make the distinction by noticing subtle differences in motion.

Since the notion of completion differs per action, a general action completion method should investigate the performance of different types of features to accommodate the various action classes. For example, for the action pick, the difference between complete and incomplete actions originates from the subtle change in body pose when holding an object, or by observing an object in the hand. On the other hand, for the action drink, the speed at which the action is performed is better able to assess the completion. We propose a method that chooses the feature(s) suitable for recognising completion from the pool of depth features using by leave-one-person-out cross validation on the training set and automatically selecting the most discriminative feature(s).

Publication:

Farnoosh Heidarivincheh, Majid Mirmehdi and Dima Damen (2016). Beyond Action Recognition: Action Completion in RGB-D Data. British Machine Vision Conference, York, UK. BMVC Proceedings, PDF, Abstract [Note: Fig3 updated]

Dataset:

Full dataset (RGB + Depth + Skeleton) [33GB] available at: http://dx.doi.org/10.5523/bris.66qry08cv1fj1eunwxwob3fjz

Sample Results:

Action Completion - Results

Fig 3 updated

Following the BMVC 2016 publications, results in Fig 3 needed updating. Please find below the new figure. Note that these updated results do not affect the paper's arguments.