Human activity recognition by fusion of RGB, depth, and skeletal data

Pushpajit Khaire; Javed Imran; Praveen Kumar

doi:10.1007/978-981-10-7895-8_32

Human activity recognition by fusion of RGB, depth, and skeletal data

Pushpajit Khaire^*, Javed Imran, Praveen Kumar

^*Corresponding author for this work

Department of Electrical & Electronic Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

17 Citations (Scopus)

Abstract

A significant increase in research of human activity recognition can be seen in recent years due to availability of low-cost RGB-D sensors and advancement of deep learning algorithms. In this paper, we augmented our previous work on human activity recognition (Imran et al., IEEE international conference on advances in computing, communications, and informatics (ICACCI), 2016) [1] by incorporating skeletal data for fusion. Three main approaches are used to fuse skeletal data with RGB, depth data, and the results are compared with each other. A challenging UTD-MHAD activity recognition dataset with intraclass variations, comprising of twenty-seven activities, is used for testing and experimentation. Proposed fusion results in accuracy of 95.38% (nearly 4% improvement over previous work), and it also justifies the fact that recognition improves with an increase in number of evidences in support.

Original language	English
Title of host publication	Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017
Editors	Bidyut B. Chaudhuri, Mohan S. Kankanhalli, Balasubramanian Raman
Publisher	Springer Verlag
Pages	409-421
Number of pages	13
ISBN (Print)	9789811078941
DOIs	https://doi.org/10.1007/978-981-10-7895-8_32
Publication status	Published - 2018
Event	2nd International Conference on Computer Vision and Image Processing, CVIP 2017 - Roorkee, India Duration: 9 Sept 2017 → 12 Sept 2017

Publication series

Name	Advances in Intelligent Systems and Computing
Volume	703
ISSN (Print)	2194-5357

Conference

Conference	2nd International Conference on Computer Vision and Image Processing, CVIP 2017
Country/Territory	India
City	Roorkee
Period	9/09/17 → 12/09/17

Bibliographical note

Funding Information:
Acknowledgements This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon.

Publisher Copyright:
© 2018, Springer Nature Singapore Pte Ltd.

Keywords

Convolutional neural networks
Deep learning
Depth motion map
Motion history image and fusion
RGB-D sensors
Skeleton
UTD-MHAD

Access to Document

10.1007/978-981-10-7895-8_32

Handle.net

Persistent link

Cite this

Khaire, P., Imran, J., & Kumar, P. (2018). Human activity recognition by fusion of RGB, depth, and skeletal data. In B. B. Chaudhuri, M. S. Kankanhalli, & B. Raman (Eds.), Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017 (pp. 409-421). (Advances in Intelligent Systems and Computing; Vol. 703). Springer Verlag. https://doi.org/10.1007/978-981-10-7895-8_32

Khaire, Pushpajit ; Imran, Javed ; Kumar, Praveen. / Human activity recognition by fusion of RGB, depth, and skeletal data. Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017. editor / Bidyut B. Chaudhuri ; Mohan S. Kankanhalli ; Balasubramanian Raman. Springer Verlag, 2018. pp. 409-421 (Advances in Intelligent Systems and Computing).

@inproceedings{d22e1fe752a44ceb8f2b7743a13d404f,

title = "Human activity recognition by fusion of RGB, depth, and skeletal data",

abstract = "A significant increase in research of human activity recognition can be seen in recent years due to availability of low-cost RGB-D sensors and advancement of deep learning algorithms. In this paper, we augmented our previous work on human activity recognition (Imran et al., IEEE international conference on advances in computing, communications, and informatics (ICACCI), 2016) [1] by incorporating skeletal data for fusion. Three main approaches are used to fuse skeletal data with RGB, depth data, and the results are compared with each other. A challenging UTD-MHAD activity recognition dataset with intraclass variations, comprising of twenty-seven activities, is used for testing and experimentation. Proposed fusion results in accuracy of 95.38% (nearly 4% improvement over previous work), and it also justifies the fact that recognition improves with an increase in number of evidences in support.",

keywords = "Convolutional neural networks, Deep learning, Depth motion map, Motion history image and fusion, RGB-D sensors, Skeleton, UTD-MHAD",

author = "Pushpajit Khaire and Javed Imran and Praveen Kumar",

note = "Funding Information: Acknowledgements This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Publisher Copyright: {\textcopyright} 2018, Springer Nature Singapore Pte Ltd.; 2nd International Conference on Computer Vision and Image Processing, CVIP 2017 ; Conference date: 09-09-2017 Through 12-09-2017",

year = "2018",

doi = "10.1007/978-981-10-7895-8_32",

language = "English",

isbn = "9789811078941",

series = "Advances in Intelligent Systems and Computing",

publisher = "Springer Verlag",

pages = "409--421",

editor = "Chaudhuri, {Bidyut B.} and Kankanhalli, {Mohan S.} and Balasubramanian Raman",

booktitle = "Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017",

address = "Germany",

}

Khaire, P, Imran, J & Kumar, P 2018, Human activity recognition by fusion of RGB, depth, and skeletal data. in BB Chaudhuri, MS Kankanhalli & B Raman (eds), Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017. Advances in Intelligent Systems and Computing, vol. 703, Springer Verlag, pp. 409-421, 2nd International Conference on Computer Vision and Image Processing, CVIP 2017, Roorkee, India, 9/09/17. https://doi.org/10.1007/978-981-10-7895-8_32

Human activity recognition by fusion of RGB, depth, and skeletal data. / Khaire, Pushpajit; Imran, Javed; Kumar, Praveen.
Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017. ed. / Bidyut B. Chaudhuri; Mohan S. Kankanhalli; Balasubramanian Raman. Springer Verlag, 2018. p. 409-421 (Advances in Intelligent Systems and Computing; Vol. 703).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Human activity recognition by fusion of RGB, depth, and skeletal data

AU - Khaire, Pushpajit

AU - Imran, Javed

AU - Kumar, Praveen

N1 - Funding Information: Acknowledgements This research was supported by Science and Engineering Research Board (SERB) under project no. ECR/2016/000387, in cooperation with the Department of Science & Technology (DST), Government of India. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of DST-SERB or the Government of India. The DST-SERB or Government of India is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. Publisher Copyright: © 2018, Springer Nature Singapore Pte Ltd.

PY - 2018

Y1 - 2018

N2 - A significant increase in research of human activity recognition can be seen in recent years due to availability of low-cost RGB-D sensors and advancement of deep learning algorithms. In this paper, we augmented our previous work on human activity recognition (Imran et al., IEEE international conference on advances in computing, communications, and informatics (ICACCI), 2016) [1] by incorporating skeletal data for fusion. Three main approaches are used to fuse skeletal data with RGB, depth data, and the results are compared with each other. A challenging UTD-MHAD activity recognition dataset with intraclass variations, comprising of twenty-seven activities, is used for testing and experimentation. Proposed fusion results in accuracy of 95.38% (nearly 4% improvement over previous work), and it also justifies the fact that recognition improves with an increase in number of evidences in support.

AB - A significant increase in research of human activity recognition can be seen in recent years due to availability of low-cost RGB-D sensors and advancement of deep learning algorithms. In this paper, we augmented our previous work on human activity recognition (Imran et al., IEEE international conference on advances in computing, communications, and informatics (ICACCI), 2016) [1] by incorporating skeletal data for fusion. Three main approaches are used to fuse skeletal data with RGB, depth data, and the results are compared with each other. A challenging UTD-MHAD activity recognition dataset with intraclass variations, comprising of twenty-seven activities, is used for testing and experimentation. Proposed fusion results in accuracy of 95.38% (nearly 4% improvement over previous work), and it also justifies the fact that recognition improves with an increase in number of evidences in support.

KW - Convolutional neural networks

KW - Deep learning

KW - Depth motion map

KW - Motion history image and fusion

KW - RGB-D sensors

KW - Skeleton

KW - UTD-MHAD

UR - http://www.scopus.com/inward/record.url?scp=85045671460&partnerID=8YFLogxK

U2 - 10.1007/978-981-10-7895-8_32

DO - 10.1007/978-981-10-7895-8_32

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85045671460

SN - 9789811078941

T3 - Advances in Intelligent Systems and Computing

SP - 409

EP - 421

BT - Proceedings of 2nd International Conference on Computer Vision and Image Processing - CVIP 2017

A2 - Chaudhuri, Bidyut B.

A2 - Kankanhalli, Mohan S.

A2 - Raman, Balasubramanian

PB - Springer Verlag

T2 - 2nd International Conference on Computer Vision and Image Processing, CVIP 2017

Y2 - 9 September 2017 through 12 September 2017

ER -

Human activity recognition by fusion of RGB, depth, and skeletal data

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this