Our previous work investigated the utility of audio as a stand-alone modality for action recognition. Whilst audio is expectedly less informative than video, we demonstrated that is very feasible to create an effective audio-alone model. Combining this result with other work demonstrating that audio is an informative modality for multi-modal action recognition, this creates a strong justification for further work investigating the utility of audio-visual action recognition, as well as exploring audio-visual fusion in a wider context, such as representation learning.

 

The self-supervised approach would consider audio-visual fusion in the context of representation learning. In representation learning a contrastive loss is used to learn a representation where data points of the same class are similar to each other, and those of different classes are dissimilar. Such a representation is learnt in a self-supervised manner where the data labels are not known. This is done by creating two examples of the same data-point, where one is a transformed version of the other (positives). A model may then learn a representation such that these examples are similar, and pairings that are not of the same data-point (negatives) are dissimilar. This self-supervised formulation can be extended to the audio-visual setting where a positive example is an audio-visual pairing, such that a model would learn similar representations for both the audio and visual components of a video, and dissimilar representations for audio-visual pairings that are from different video clips. Through the introduction of multiple modalities, we can enhance the representation that is learnt through this process, with the aim of learning representations that are informative for down-stream tasks. In particular, we are interested in the utility of audio-visual representation learning to enhance the performance of egocentric action recognition.