HIGS: Hand interaction guidance system

Yao Lu; Walterio Mayol-Cuevas

doi:10.1109/ISMAR-Adjunct.2019.000-7

HIGS: Hand interaction guidance system

Yao Lu
, Walterio Mayol-Cuevas

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

6 Citations (Scopus)

Abstract

In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.

Original language	English
Title of host publication	Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	376-381
Number of pages	6
ISBN (Electronic)	9781728147659
DOIs	https://doi.org/10.1109/ISMAR-Adjunct.2019.000-7
Publication status	Published - Oct 2019
Event	18th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 - Beijing, China Duration: 14 Oct 2019 → 18 Oct 2019

Publication series

Name	Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019

Conference

Conference	18th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019
Country/Territory	China
City	Beijing
Period	14/10/19 → 18/10/19

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

Keywords

Augmented Reality
HCI system design and evaluation methods
Human-centered computing

Access to Document

10.1109/ISMAR-Adjunct.2019.000-7

Handle.net

Persistent link

Cite this

Lu, Y., & Mayol-Cuevas, W. (2019). HIGS: Hand interaction guidance system. In Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 (pp. 376-381). Article 8951892 (Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ISMAR-Adjunct.2019.000-7

Lu, Yao ; Mayol-Cuevas, Walterio. / HIGS : Hand interaction guidance system. Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019. Institute of Electrical and Electronics Engineers (IEEE), 2019. pp. 376-381 (Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019).

@inproceedings{6ee5bc8148cc4bdd96d38aff219e7fcd,

title = "HIGS: Hand interaction guidance system",

abstract = "In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.",

keywords = "Augmented Reality, HCI system design and evaluation methods, Human-centered computing",

author = "Yao Lu and Walterio Mayol-Cuevas",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 18th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019 ; Conference date: 14-10-2019 Through 18-10-2019",

year = "2019",

month = oct,

doi = "10.1109/ISMAR-Adjunct.2019.000-7",

language = "English",

series = "Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "376--381",

booktitle = "Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019",

address = "United States",

}

Lu, Y & Mayol-Cuevas, W 2019, HIGS: Hand interaction guidance system. in Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019., 8951892, Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019, Institute of Electrical and Electronics Engineers (IEEE), pp. 376-381, 18th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019, Beijing, China, 14/10/19. https://doi.org/10.1109/ISMAR-Adjunct.2019.000-7

HIGS: Hand interaction guidance system. / Lu, Yao; Mayol-Cuevas, Walterio.
Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019. Institute of Electrical and Electronics Engineers (IEEE), 2019. p. 376-381 8951892 (Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - HIGS

T2 - 18th IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019

AU - Lu, Yao

AU - Mayol-Cuevas, Walterio

PY - 2019/10

Y1 - 2019/10

N2 - In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.

AB - In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.

KW - Augmented Reality

KW - HCI system design and evaluation methods

KW - Human-centered computing

UR - https://www.scopus.com/pages/publications/85078714517

U2 - 10.1109/ISMAR-Adjunct.2019.000-7

DO - 10.1109/ISMAR-Adjunct.2019.000-7

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85078714517

T3 - Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019

SP - 376

EP - 381

BT - Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019

PB - Institute of Electrical and Electronics Engineers (IEEE)

Y2 - 14 October 2019 through 18 October 2019

ER -

Lu Y, Mayol-Cuevas W. HIGS: Hand interaction guidance system. In Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019. Institute of Electrical and Electronics Engineers (IEEE). 2019. p. 376-381. 8951892. (Adjunct Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2019). doi: 10.1109/ISMAR-Adjunct.2019.000-7

HIGS: Hand interaction guidance system

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this