Action Modifiers: Learning from Adverbs in Instructional Videos

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

Abstract

We present a method to learn a representation for adverbs from instructional videos using weak supervision from the accompanying narrations. Key to our method is the fact that the visual representation of the adverb is highly dependant on the action to which it applies, although the same adverb will modify multiple actions in a similar way. For instance, while ‘spread quickly’ and ‘mix quickly’ will look dissimilar, we can learn a common representation that allows us to recognize both, among other actions. We formulate this as an embedding problem, and use scaled dot-product attention to learn from weakly supervised video narrations. We jointly learn adverbs as invertible transformations operating on the embedding space, so as to add or remove the effect of the adverb. As there is no prior work on weakly supervised learning of adverbs, we gather paired action-adverb annotations from a subset of the HowTo100M dataset for 6 adverbs: quickly/slowly, finely/coarsely, and partially/completely. Our method outperforms all baselines for video-to-adverb retrieval with a performance of 0.719 mAP. We also demonstrate our model’s ability to attend to the relevant video parts in order to determine the adverb for a given action.
Original languageEnglish
Title of host publicationIEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages12
Publication statusAccepted/In press - 27 Feb 2020
EventInternational Conference on Computer Vision and Pattern Recognition - Seattle, United States
Duration: 16 Jun 202018 Jun 2020
http://cvpr2020.thecvf.com/

Conference

ConferenceInternational Conference on Computer Vision and Pattern Recognition
Abbreviated titleCVPR2020
CountryUnited States
CitySeattle
Period16/06/2018/06/20
Internet address

Fingerprint Dive into the research topics of 'Action Modifiers: Learning from Adverbs in Instructional Videos'. Together they form a unique fingerprint.

  • Projects

    Cite this

    Doughty, H. R., Laptev, I., Mayol-Cuevas, W. W., & Damen, D. (Accepted/In press). Action Modifiers: Learning from Adverbs in Instructional Videos. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR) Institute of Electrical and Electronics Engineers (IEEE).