Talk, text or tag? the development of a self-annotation app for activity recognition in smart environments

Pervasive computing and, specifically, the Internet of Things aspire to deliver smart services and effortless interactions for their users. Achieving this requires making sense of multiple streams of sensor data, which becomes particularly challenging when these concern people’s activities in the real world. In this paper we describe the exploration of different approaches that allow users to self-annotate their activities in near real-time, which in turn can be used as ground-truth to develop algorithms for automated and accurate activity recognition. We offer the lessons we learnt during each design iteration of a smart-phone app and detail how we arrived at our current approach to acquiring ground-truth data ‘in the wild’. In doing so, we uncovered tensions between researchers’ data annotation requirements and users’ interaction requirements, which need equal consideration if an acceptable self-annotation solution is to be achieved. We present an ongoing user study of a hybrid approach, which supports activity logging that is appropriate to different individuals and contexts.