Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

• Type of Material: Article
• Authors: Hyeokhyen Kwon, Emory University; Gregory D. Abowd, Georgia Institute of Technology; Thomas Ploetz, Georgia Institute of Technology
• Language: English
• Date: 2021-12-01
• Publisher: MDPI
• Publication Version: Final Published Version
• Copyright Statement: © 2021 by the authors.
• License: Creative Commons Attribution 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.3390/s21248337
• Title of Journal or Parent Work: SENSORS
• Volume: 21
• Issue: 24
• Start Page: 8337
• End Page: 8337
• Grant/Funding Information: The computational needs for this research are supported in part by Oracle Cloud credits and related resources provided by the Oracle for Research program. Mr Kwon is supported in part by the Samsung Scholarship Program.
• Keywords: Engineering, Electrical & Electronic; Engineering; Physical Sciences; Technology; virtual IMU data; human activity recognition; Chemistry; Science & Technology; CLASSIFICATION; Instruments & Instrumentation; deep learning; Chemistry, Analytical; PATTERNS
• Research Categories: Education, Technology; Chemistry, General

Abstract

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.