Combining CNN streams of RGB-D and skeletal data for human activity recognition

Pushpajit Khaire; Praveen Kumar; Javed Imran

doi:10.1016/j.patrec.2018.04.035

Combining CNN streams of RGB-D and skeletal data for human activity recognition

Pushpajit Khaire^*
, Praveen Kumar
, Javed Imran

^*Corresponding author for this work

Research output: Contribution to journal › Article (Academic Journal) › peer-review

140 Citations (Scopus)

Abstract

Inspired by the success of deep learning methods, for human activity recognition based on individual vision cues, this paper presents a ConvNets based approach for activity recognition by combining multiple vision cues. Moreover, a new method of creating skeleton images, from skeleton joint sequences, representing motion information is presented in this paper. Motion representation images, namely, Motion History Image (MHI), Depth Motion Maps (DMMs) and skeleton images are constructed from RGB, depth and skeletal data of RGB-D sensor. These images are then separately trained on ConvNets and respective softmax scores are fused at the decision level. The combination of these distinct vision cues, leads to complete utilization of data, available from RGB-D sensor. To evaluate the effectiveness of the proposed 5-CNNs approach, we conduct our experiments on three well known and challenging RGB-D datasets, CAD-60, SBU Kinect interaction and UTD-MHAD. Results show that the proposed approach of combining multiple cues by means of decision level fusion is competitive with other state of the art methods.

Original language	English
Pages (from-to)	107-116
Number of pages	10
Journal	Pattern Recognition Letters
Volume	115
DOIs	https://doi.org/10.1016/j.patrec.2018.04.035
Publication status	Published - 1 Nov 2018

Bibliographical note

Funding Information:
This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387 , in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon.

Funding Information:
This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon.

Publisher Copyright:
© 2018 Elsevier B.V.

Keywords

Convolutional neural networks
Deep learning
Depth motion map
Motion history image and fusion
RGB-D sensors
Skeleton
UTD-MHAD

Access to Document

10.1016/j.patrec.2018.04.035

Handle.net

Persistent link

Cite this

@article{f412650becfd4a99927e70147b36eed6,

title = "Combining CNN streams of RGB-D and skeletal data for human activity recognition",

abstract = "Inspired by the success of deep learning methods, for human activity recognition based on individual vision cues, this paper presents a ConvNets based approach for activity recognition by combining multiple vision cues. Moreover, a new method of creating skeleton images, from skeleton joint sequences, representing motion information is presented in this paper. Motion representation images, namely, Motion History Image (MHI), Depth Motion Maps (DMMs) and skeleton images are constructed from RGB, depth and skeletal data of RGB-D sensor. These images are then separately trained on ConvNets and respective softmax scores are fused at the decision level. The combination of these distinct vision cues, leads to complete utilization of data, available from RGB-D sensor. To evaluate the effectiveness of the proposed 5-CNNs approach, we conduct our experiments on three well known and challenging RGB-D datasets, CAD-60, SBU Kinect interaction and UTD-MHAD. Results show that the proposed approach of combining multiple cues by means of decision level fusion is competitive with other state of the art methods.",

keywords = "Convolutional neural networks, Deep learning, Depth motion map, Motion history image and fusion, RGB-D sensors, Skeleton, UTD-MHAD",

author = "Pushpajit Khaire and Praveen Kumar and Javed Imran",

note = "Funding Information: This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387 , in cooperation with the Department of Science \& Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Funding Information: This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science \& Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Publisher Copyright: {\textcopyright} 2018 Elsevier B.V.",

year = "2018",

month = nov,

day = "1",

doi = "10.1016/j.patrec.2018.04.035",

language = "English",

volume = "115",

pages = "107--116",

journal = "Pattern Recognition Letters",

issn = "0167-8655",

publisher = "Elsevier B.V.",

}

TY - JOUR

T1 - Combining CNN streams of RGB-D and skeletal data for human activity recognition

AU - Khaire, Pushpajit

AU - Kumar, Praveen

AU - Imran, Javed

N1 - Funding Information: This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387 , in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Funding Information: This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Publisher Copyright: © 2018 Elsevier B.V.

PY - 2018/11/1

Y1 - 2018/11/1

N2 - Inspired by the success of deep learning methods, for human activity recognition based on individual vision cues, this paper presents a ConvNets based approach for activity recognition by combining multiple vision cues. Moreover, a new method of creating skeleton images, from skeleton joint sequences, representing motion information is presented in this paper. Motion representation images, namely, Motion History Image (MHI), Depth Motion Maps (DMMs) and skeleton images are constructed from RGB, depth and skeletal data of RGB-D sensor. These images are then separately trained on ConvNets and respective softmax scores are fused at the decision level. The combination of these distinct vision cues, leads to complete utilization of data, available from RGB-D sensor. To evaluate the effectiveness of the proposed 5-CNNs approach, we conduct our experiments on three well known and challenging RGB-D datasets, CAD-60, SBU Kinect interaction and UTD-MHAD. Results show that the proposed approach of combining multiple cues by means of decision level fusion is competitive with other state of the art methods.

AB - Inspired by the success of deep learning methods, for human activity recognition based on individual vision cues, this paper presents a ConvNets based approach for activity recognition by combining multiple vision cues. Moreover, a new method of creating skeleton images, from skeleton joint sequences, representing motion information is presented in this paper. Motion representation images, namely, Motion History Image (MHI), Depth Motion Maps (DMMs) and skeleton images are constructed from RGB, depth and skeletal data of RGB-D sensor. These images are then separately trained on ConvNets and respective softmax scores are fused at the decision level. The combination of these distinct vision cues, leads to complete utilization of data, available from RGB-D sensor. To evaluate the effectiveness of the proposed 5-CNNs approach, we conduct our experiments on three well known and challenging RGB-D datasets, CAD-60, SBU Kinect interaction and UTD-MHAD. Results show that the proposed approach of combining multiple cues by means of decision level fusion is competitive with other state of the art methods.

KW - Convolutional neural networks

KW - Deep learning

KW - Depth motion map

KW - Motion history image and fusion

KW - RGB-D sensors

KW - Skeleton

KW - UTD-MHAD

UR - https://www.scopus.com/pages/publications/85048732871

U2 - 10.1016/j.patrec.2018.04.035

DO - 10.1016/j.patrec.2018.04.035

M3 - Article (Academic Journal)

AN - SCOPUS:85048732871

SN - 0167-8655

VL - 115

SP - 107

EP - 116

JO - Pattern Recognition Letters

JF - Pattern Recognition Letters

ER -

Combining CNN streams of RGB-D and skeletal data for human activity recognition

Abstract

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this