Compositionality in Vision and Language

Add to your list(s) Download to your calendar using vCal

• Siddharth Narayanaswamy, School of Electrical and Computer Engineering, Purdue University, USA
• Monday 03 June 2013, 16:00-17:00
• UG05, Learning Centre.

If you have a question about this talk, please contact Leandro Minku.

This talk has been canceled/deleted

People often perceive and/or infer things about the world by incorporating information available from multiple sources in the context of world knowledge. This ability to integrate information across modalities is the focus of my research, especially, the integration between vision and language. I endeavour to build systems that exhibit this ability, and in the process, gain further insight into the processes of perception and inference as exhibited by humans.

One line of research involves reasoning about the physical structure of composable entities, involving integration within vision, and across vision and language through the medium of robotics. I show how even information extracted from unreliable sources, such as feature detectors, in the presence of occlusion, is useful when reasoned about in the context of other low-level sources, natural-language descriptions of the entities, and basic world knowledge.

Another line of research shows how language, and the compositional structure of events and sentences, interplays with the underlying tracking mechanisms in video action recognition. Such a combined framework allows for video retrieval, video description, and focus-of-attention in multi-activity videos.

I further explore compositionality in a first-of-its-kind study where people in an fMRI machine are shown videos depicting activities that correspond to unique sentential descriptions. By classifying brain- activity with different subsets of the sentential structure, first independently and then jointly, I show that brain-activity patterns reflect compositionality in sentence structure as the composition of independent classifications matches those obtained jointly.

PDF version of abstract with papers and links: http://www.cs.bham.ac.uk/research/projects/cogaff/visitors/siddharth-abstract.pdf

——

Please note that coffee will be served at 3:15 at the coffee room (School of Computer Science), followed by the one-hour seminar at 4pm.

This talk is part of the Artificial Intelligence and Natural Computation seminars series.

This talk is included in these lists:

This talk is not included in any other list

Note that ex-directory lists are not shown.