Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos

Tanqiu Qiao; Qianhui Men; Frederick W.B. Li; Yoshiki Kubotani; Shigeo Morishima; Hubert P.H. Shum

doi:10.1007/978-3-031-19772-7_28

Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos

Tanqiu Qiao, Qianhui Men, Frederick W.B. Li, Yoshiki Kubotani, Shigeo Morishima, Hubert P.H. Shum^*

^*Corresponding author for this work

School of Engineering Mathematics and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

10 Citations (Scopus)

Abstract

Human-Object Interaction (HOI) recognition in videos is important for analyzing human activity. Most existing work focusing on visual features usually suffer from occlusion in the real-world scenarios. Such a problem will be further complicated when multiple people and objects are involved in HOIs. Consider that geometric features such as human pose and object position provide meaningful information to understand HOIs, we argue to combine the benefits of both visual and geometric features in HOI recognition, and propose a novel Two-level Geometric feature-informed Graph Convolutional Network (2G-GCN ). The geometric-level graph models the interdependency between geometric features of humans and objects, while the fusion-level graph further fuses them with visual features of humans and objects. To demonstrate the novelty and effectiveness of our method in challenging scenarios, we propose a new multi-person HOI dataset (MPHOI-72 ). Extensive experiments on MPHOI-72 (multi-person HOI), CAD-120 (single-human HOI) and Bimanual Actions (two-hand HOI) datasets demonstrate our superior performance compared to state-of-the-arts.

Original language	English
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	474-491
Number of pages	18
ISBN (Electronic)	9783031197727
ISBN (Print)	9783031197710
DOIs	https://doi.org/10.1007/978-3-031-19772-7_28
Publication status	Published - 28 Oct 2022
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: 23 Oct 2022 → 27 Oct 2022 https://eccv2022.ecva.net/

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13664
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	23/10/22 → 27/10/22
Internet address	https://eccv2022.ecva.net/

Bibliographical note

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

Feature fusion
Graph convolution neural networks
Human-object interaction
Multi-person interaction

Access to Document

10.1007/978-3-031-19772-7_28

Handle.net

Persistent link

Cite this

Qiao, T., Men, Q., Li, F. W. B., Kubotani, Y., Morishima, S., & Shum, H. P. H. (2022). Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings (pp. 474-491). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13664). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-19772-7_28

Qiao, Tanqiu ; Men, Qianhui ; Li, Frederick W.B. et al. / Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos. Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. pp. 474-491 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5241cfcebe364f53aa9f5df8d7364a76,

title = "Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos",

abstract = "Human-Object Interaction (HOI) recognition in videos is important for analyzing human activity. Most existing work focusing on visual features usually suffer from occlusion in the real-world scenarios. Such a problem will be further complicated when multiple people and objects are involved in HOIs. Consider that geometric features such as human pose and object position provide meaningful information to understand HOIs, we argue to combine the benefits of both visual and geometric features in HOI recognition, and propose a novel Two-level Geometric feature-informed Graph Convolutional Network (2G-GCN ). The geometric-level graph models the interdependency between geometric features of humans and objects, while the fusion-level graph further fuses them with visual features of humans and objects. To demonstrate the novelty and effectiveness of our method in challenging scenarios, we propose a new multi-person HOI dataset (MPHOI-72 ). Extensive experiments on MPHOI-72 (multi-person HOI), CAD-120 (single-human HOI) and Bimanual Actions (two-hand HOI) datasets demonstrate our superior performance compared to state-of-the-arts.",

keywords = "Feature fusion, Graph convolution neural networks, Human-object interaction, Multi-person interaction",

author = "Tanqiu Qiao and Qianhui Men and Li, {Frederick W.B.} and Yoshiki Kubotani and Shigeo Morishima and Shum, {Hubert P.H.}",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

month = oct,

day = "28",

doi = "10.1007/978-3-031-19772-7_28",

language = "English",

isbn = "9783031197710",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "474--491",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings",

url = "https://eccv2022.ecva.net/",

}

Qiao, T, Men, Q, Li, FWB, Kubotani, Y, Morishima, S & Shum, HPH 2022, Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13664, Springer Science and Business Media Deutschland GmbH, pp. 474-491, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 23/10/22. https://doi.org/10.1007/978-3-031-19772-7_28

Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos. / Qiao, Tanqiu; Men, Qianhui; Li, Frederick W.B. et al.
Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. p. 474-491 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13664).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos

AU - Qiao, Tanqiu

AU - Men, Qianhui

AU - Li, Frederick W.B.

AU - Kubotani, Yoshiki

AU - Morishima, Shigeo

AU - Shum, Hubert P.H.

PY - 2022/10/28

Y1 - 2022/10/28

N2 - Human-Object Interaction (HOI) recognition in videos is important for analyzing human activity. Most existing work focusing on visual features usually suffer from occlusion in the real-world scenarios. Such a problem will be further complicated when multiple people and objects are involved in HOIs. Consider that geometric features such as human pose and object position provide meaningful information to understand HOIs, we argue to combine the benefits of both visual and geometric features in HOI recognition, and propose a novel Two-level Geometric feature-informed Graph Convolutional Network (2G-GCN ). The geometric-level graph models the interdependency between geometric features of humans and objects, while the fusion-level graph further fuses them with visual features of humans and objects. To demonstrate the novelty and effectiveness of our method in challenging scenarios, we propose a new multi-person HOI dataset (MPHOI-72 ). Extensive experiments on MPHOI-72 (multi-person HOI), CAD-120 (single-human HOI) and Bimanual Actions (two-hand HOI) datasets demonstrate our superior performance compared to state-of-the-arts.

AB - Human-Object Interaction (HOI) recognition in videos is important for analyzing human activity. Most existing work focusing on visual features usually suffer from occlusion in the real-world scenarios. Such a problem will be further complicated when multiple people and objects are involved in HOIs. Consider that geometric features such as human pose and object position provide meaningful information to understand HOIs, we argue to combine the benefits of both visual and geometric features in HOI recognition, and propose a novel Two-level Geometric feature-informed Graph Convolutional Network (2G-GCN ). The geometric-level graph models the interdependency between geometric features of humans and objects, while the fusion-level graph further fuses them with visual features of humans and objects. To demonstrate the novelty and effectiveness of our method in challenging scenarios, we propose a new multi-person HOI dataset (MPHOI-72 ). Extensive experiments on MPHOI-72 (multi-person HOI), CAD-120 (single-human HOI) and Bimanual Actions (two-hand HOI) datasets demonstrate our superior performance compared to state-of-the-arts.

KW - Feature fusion

KW - Graph convolution neural networks

KW - Human-object interaction

KW - Multi-person interaction

UR - http://www.scopus.com/inward/record.url?scp=85142736903&partnerID=8YFLogxK

UR - https://eccv2022.ecva.net/

U2 - 10.1007/978-3-031-19772-7_28

DO - 10.1007/978-3-031-19772-7_28

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85142736903

SN - 9783031197710

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 474

EP - 491

BT - Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th European Conference on Computer Vision, ECCV 2022

Y2 - 23 October 2022 through 27 October 2022

ER -

Qiao T, Men Q, Li FWB, Kubotani Y, Morishima S, Shum HPH. Geometric Features Informed Multi-person Human-Object Interaction Recognition in Videos. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 474-491. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-19772-7_28