Deep learning and RGB-D based human action, human–human and human–object interaction recognition: A survey

Pushpajit Khaire; Praveen Kumar

doi:10.1016/j.jvcir.2022.103531

Deep learning and RGB-D based human action, human–human and human–object interaction recognition: A survey

Pushpajit Khaire^*, Praveen Kumar

^*Corresponding author for this work

Department of Electrical & Electronic Engineering

Research output: Contribution to journal › Article (Academic Journal) › peer-review

34 Citations (Scopus)

Abstract

Human activity recognition is one of the most studied topics in the field of computer vision. In recent years, with the availability of RGB-D sensors and powerful deep learning techniques, research on human activity recognition has gained momentum. From simple human atomic actions, the research has advanced towards recognizing more complex human activities using RGB-D data. This paper presents a comprehensive survey of the advanced deep learning based recognition methods and categorizes them in human atomic action, human–human interaction, human–object interaction. The reviewed methods are further classified based on the individual modality used for recognition i.e. RGB based, depth based, skeleton based, and hybrid. We also review and categorize recent challenging RGB-D datasets for the same. In addition, the paper also briefly reviews RGB-D datasets and methods for online activity recognition. The paper concludes with a discussion on limitations, challenges, and recent trends for promising future directions.

Original language	English
Article number	103531
Journal	Journal of Visual Communication and Image Representation
Volume	86
Early online date	7 May 2022
DOIs	https://doi.org/10.1016/j.jvcir.2022.103531
Publication status	Published - Jul 2022

Bibliographical note

Funding Information:
This research was supported by Science and Engineering Research Board (SERB), India under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. We are also thankful to all the reviewers for their valuable comments and suggestions to improve the scientific value of the paper.

Publisher Copyright:
© 2022 Elsevier Inc.

Keywords

CNN
Deep learning
Fusion
GCN
Human action recognition
Human–human interaction
Human–object interaction
LSTM
Multi-modality
RGB-D sensors
Skeleton

Access to Document

10.1016/j.jvcir.2022.103531

Handle.net

Persistent link

Cite this

@article{ac7392982536472ea6ae843fb477ac41,

title = "Deep learning and RGB-D based human action, human–human and human–object interaction recognition: A survey",

abstract = "Human activity recognition is one of the most studied topics in the field of computer vision. In recent years, with the availability of RGB-D sensors and powerful deep learning techniques, research on human activity recognition has gained momentum. From simple human atomic actions, the research has advanced towards recognizing more complex human activities using RGB-D data. This paper presents a comprehensive survey of the advanced deep learning based recognition methods and categorizes them in human atomic action, human–human interaction, human–object interaction. The reviewed methods are further classified based on the individual modality used for recognition i.e. RGB based, depth based, skeleton based, and hybrid. We also review and categorize recent challenging RGB-D datasets for the same. In addition, the paper also briefly reviews RGB-D datasets and methods for online activity recognition. The paper concludes with a discussion on limitations, challenges, and recent trends for promising future directions.",

keywords = "CNN, Deep learning, Fusion, GCN, Human action recognition, Human–human interaction, Human–object interaction, LSTM, Multi-modality, RGB-D sensors, Skeleton",

author = "Pushpajit Khaire and Praveen Kumar",

note = "Funding Information: This research was supported by Science and Engineering Research Board (SERB), India under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. We are also thankful to all the reviewers for their valuable comments and suggestions to improve the scientific value of the paper. Publisher Copyright: {\textcopyright} 2022 Elsevier Inc.",

year = "2022",

month = jul,

doi = "10.1016/j.jvcir.2022.103531",

language = "English",

volume = "86",

journal = "Journal of Visual Communication and Image Representation",

issn = "1047-3203",

publisher = "Academic Press",

}

TY - JOUR

T1 - Deep learning and RGB-D based human action, human–human and human–object interaction recognition

T2 - A survey

AU - Khaire, Pushpajit

AU - Kumar, Praveen

N1 - Funding Information: This research was supported by Science and Engineering Research Board (SERB), India under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. We are also thankful to all the reviewers for their valuable comments and suggestions to improve the scientific value of the paper. Publisher Copyright: © 2022 Elsevier Inc.

PY - 2022/7

Y1 - 2022/7

N2 - Human activity recognition is one of the most studied topics in the field of computer vision. In recent years, with the availability of RGB-D sensors and powerful deep learning techniques, research on human activity recognition has gained momentum. From simple human atomic actions, the research has advanced towards recognizing more complex human activities using RGB-D data. This paper presents a comprehensive survey of the advanced deep learning based recognition methods and categorizes them in human atomic action, human–human interaction, human–object interaction. The reviewed methods are further classified based on the individual modality used for recognition i.e. RGB based, depth based, skeleton based, and hybrid. We also review and categorize recent challenging RGB-D datasets for the same. In addition, the paper also briefly reviews RGB-D datasets and methods for online activity recognition. The paper concludes with a discussion on limitations, challenges, and recent trends for promising future directions.

AB - Human activity recognition is one of the most studied topics in the field of computer vision. In recent years, with the availability of RGB-D sensors and powerful deep learning techniques, research on human activity recognition has gained momentum. From simple human atomic actions, the research has advanced towards recognizing more complex human activities using RGB-D data. This paper presents a comprehensive survey of the advanced deep learning based recognition methods and categorizes them in human atomic action, human–human interaction, human–object interaction. The reviewed methods are further classified based on the individual modality used for recognition i.e. RGB based, depth based, skeleton based, and hybrid. We also review and categorize recent challenging RGB-D datasets for the same. In addition, the paper also briefly reviews RGB-D datasets and methods for online activity recognition. The paper concludes with a discussion on limitations, challenges, and recent trends for promising future directions.

KW - CNN

KW - Deep learning

KW - Fusion

KW - GCN

KW - Human action recognition

KW - Human–human interaction

KW - Human–object interaction

KW - LSTM

KW - Multi-modality

KW - RGB-D sensors

KW - Skeleton

UR - http://www.scopus.com/inward/record.url?scp=85129961286&partnerID=8YFLogxK

U2 - 10.1016/j.jvcir.2022.103531

DO - 10.1016/j.jvcir.2022.103531

M3 - Article (Academic Journal)

AN - SCOPUS:85129961286

SN - 1047-3203

VL - 86

JO - Journal of Visual Communication and Image Representation

JF - Journal of Visual Communication and Image Representation

M1 - 103531

ER -

Deep learning and RGB-D based human action, human–human and human–object interaction recognition: A survey

Abstract

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this