Enabling AI systems to possess human level intelligence is a highly desirable feature sought after by researchers in both the academia and the industry. Learning from first person perspective has been shown to aid the process of learning in humans. Therefore, it holds the potential to allow AI system to become smarter. Moreover, humans utilise their different sensory inputs as clues on daily basis for different purposes such navigating their environment and learning new skills. As a result, there have been a plethora of multimodal egocentric datasets published in recent years such as EPIC-KITCHENS, EGO4D and EgoExo4D. These datasets introduce videos recorded from first person perspective accompanied with modalities such as text in the form of narrations and expert commentary and both mono and stereo audio.  

This project will focus on training AI systems on multimodal computer vision datasets to address unsolved challenges in related domains such as sound source localisation in the audio-visual correspondence domain. Furthermore, this project will explore the potential of using additional modalities, such as text, to help in solving such problems and will introduce ablation studies to determine the value of the different modalities in solving the challenges of interest.