Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization - CRISTAL-MAGNET
Conference Papers Year : 2023

Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization

Abstract

In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, "oil" mixed with "salt" and "pepper" could later be referred to as a "mixture". Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression "mixture", and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as visual-linguistic ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of visual-linguistic ambiguity, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization as compared to a strong single-task baseline.
Fichier principal
Vignette du fichier
emnlp23impress.pdf (4.64 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04259861 , version 1 (26-10-2023)

Identifiers

  • HAL Id : hal-04259861 , version 1

Cite

Cennet Oguz, Pascal Denis, Emmanuel Vincent, Simon Ostermann, Josef van Genabith. Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization. 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023, Singapore, Singapore. ⟨hal-04259861⟩
145 View
97 Download

Share

More