Skip to main content
SpringerLink
Log in

Detecting Human Activities Based on a Multimodal Sensor Data Set Using a Bidirectional Long Short-Term Memory Model: A Case Study

  • Chapter
  • First Online:
Challenges and Trends in Multimodal Fall Detection for Healthcare

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 273))

Abstract

Human falls are one of the leading causes of fatal unintentional injuries worldwide. Falls result in a direct financial cost to health systems, and indirectly, to society’s productivity. Unsurprisingly, human fall detection and prevention is a major focus of health research. In this chapter, we present and evaluate several bidirectional long short-term memory (Bi-LSTM) models using a data set provided by the Challenge UP competition. The main goal of this study is to detect 12 human daily activities (six daily human activities, five falls, and one post-fall activity) derived from multi-modal data sources - wearable sensors, ambient sensors, and vision devices. Our proposed Bi-LSTM model leverages data from accelerometer and gyroscope sensors located at the ankle, right pocket, belt, and neck of the subject. We utilize a grid search technique to evaluate variations of the Bi-LSTM model and identify a configuration that presents the best results. The best Bi-LSTM model achieved good results for precision and f1-score, 43.30 and 38.50%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Tunesia)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 85.59
Price includes VAT (Tunesia)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 99.99
Price excludes VAT (Tunesia)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
EUR 99.99
Price excludes VAT (Tunesia)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The entire code is available for download at https://github.com/GutoL/ChallengeUP.

  2. 2.

    http://keras.io/.

  3. 3.

    https://www.tensorflow.org/.

References

  1. World Health Organization: Falls, Fact Sheet. http://www.who.int/news-room/fact-sheets/detail/falls (2018). Accessed 08 Oct 2008

  2. Organization, W.H.: WHO Global Report on Falls: Prevention in Older Age. World Health Organization (2007)

    Google Scholar 

  3. Scheffer, A.C., Schuurmans, M.J., Van Dijk, N., Van Der Hooft, T., De Rooij, S.E.: Fear of falling: measurement strategy, prevalence, risk factors and consequences among older persons. Age Ageing 37(1), 19 (2008)

    Article  Google Scholar 

  4. Igual, R., Medrano, C., Plaza, I.: Challenges, issues and trends in fall detection systems. Biomed. Eng. Online 12(1), 66 (2013)

    Article  Google Scholar 

  5. Brownsell, S., Hawley, M.S.: Automatic fall detectors and the fear of falling. J. Telemed. Telecare 10(5), 262 (2004)

    Article  Google Scholar 

  6. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 689–696, (2011)

    Google Scholar 

  7. Rouast, P.V., Adam, M., Chiong, R.: Deep learning for human affect recognition: insights and new developments. IEEE Trans. Affect. Comput. (2019)

    Google Scholar 

  8. Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423 (2019)

    Article  Google Scholar 

  9. Nweke, H.F., Teh, Y.W., Mujtaba, G., Al-Garadi, M.A.: Data fusion and multiple classifier systems for human activity detection and health monitoring: review and open research directions. Inf. Fusion 46, 147 (2019)

    Article  Google Scholar 

  10. Chen, C., Jafari, R., Kehtarnavaz, N.: A survey of depth and inertial sensor fusion for human action recognition. Multimed. Tools Appl. 76(3), 4405 (2017)

    Article  Google Scholar 

  11. Ordóñez, F., Roggen, D.: Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)

    Article  Google Scholar 

  12. Bengio, Y., Simard, P., Frasconi, P., et al.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157 (1994)

    Article  Google Scholar 

  13. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain., Fuzziness Knowl.-Based Syst. 6(02), 107 (1998)

    Article  Google Scholar 

  14. Lin, H.Y., Hsueh, Y.L., Lie, W.N.: Abnormal event detection using microsoft kinect in a smart home. In: 2016 International Computer Symposium (ICS). IEEE, pp. 285–289 (2016)

    Google Scholar 

  15. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735 (1997)

    Article  Google Scholar 

  16. Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: International Conference on Machine Learning, pp. 2342–2350 (2015)

    Google Scholar 

  17. Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: Lstm: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222 (2017)

    Article  MathSciNet  Google Scholar 

  18. Understanding lstm and its diagrams. https://medium.com/mlreview/understanding-lstm-and-its-diagrams-37e2f46f1714 (2016). Accessed Aug 2018

  19. Zhao, R., Yan, R., Wang, J., Mao, K.: Learning to monitor machine health with convolutional bi-directional lstm networks. Sensors 17(2), 273 (2017)

    Article  Google Scholar 

  20. Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep learning, vol. 1. MIT press Cambridge (2016)

    Google Scholar 

  21. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673 (1997)

    Article  Google Scholar 

  22. Challenge up website. Available at: https://sites.google.com/up.edu.mx/challenge-up-2019/overview?authuser=0 (2019). Last access Apr 2019

  23. Martínez-Villaseñor, L., Ponce, H., Brieva, J., Moya-Albor, E., Núñez-Martínez, J., Peñafort-Asturiano, C.: Up-fall detection dataset: a multimodal approach. Sensors 19(9), 1988 (2019)

    Article  Google Scholar 

  24. Zhao, K., Chu, W.S., Zhang, H.: Deep region and multi-label learning for facial action unit detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3391–3399 (2016)

    Google Scholar 

  25. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)

  26. Berg, R.V.D., Kipf, T.N., Welling, M.: Graph convolutional matrix completion, arXiv preprint arXiv:1706.02263 (2017)

  27. Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M.: Deeplabcut: Markerless Pose Estimation of User-Defined Body Parts with Deep Learning. Nature Publishing Group, Tech. rep. (2018)

    Article  Google Scholar 

  28. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(Feb), 281 (2012)

    Google Scholar 

  29. Li, J., Zhang, C., Li, Z.: Battlefield target identification based on improved grid-search svm classifier. In 2009 International Conference on Computational Intelligence and Software Engineering, pp. 1–4. IEEE (2009)

    Google Scholar 

  30. Hesterman, J.Y., Caucci, L., Kupinski, M.A., Barrett, H.H., Furenlid, L.R.: Maximum-likelihood estimation with a contracting-grid search algorithm. IEEE Trans. Nucl. Sci. 57(3), 1077 (2010)

    Article  Google Scholar 

  31. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01578 (2016)

  32. Akay, M.F.: Support vector machines combined with feature selection for breast cancer diagnosis. Expert. Syst. Appl. 36(2), 3240 (2009)

    Article  Google Scholar 

  33. Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recognit. Lett. 119, 3 (2019)

    Article  Google Scholar 

  34. Vepakomma, P., De, D., Das, S.K., Bhansali, S.: A-wristocracy: deep learning on wrist-worn sensing for recognition of user complex activities. In: 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), pp. 1–6. IEEE (2015)

    Google Scholar 

  35. Ding, L., Fang, W., Luo, H., Love, P.E., Zhong, B., Ouyang, X.: A deep hybrid learning model to detect unsafe behavior: integrating convolution neural networks and long short-term memory. Autom. Constr. 86, 118 (2018)

    Article  Google Scholar 

  36. Núñez, J.C., Cabido, R., Pantrigo, J.J., Montemayor, A.S., Vélez, J.F.: Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recognit. 76, 80 (2018)

    Article  Google Scholar 

  37. Altun, K., Barshan, B.: Human activity recognition using inertial/magnetic sensor units. In: International Workshop on Human Behavior Understanding, pp. 38–51. Springer (2010)

    Google Scholar 

  38. Ermes, M., Pärkkä, J., Mäntyjärvi, J., Korhonen, I.: Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions. IEEE Trans. Inf. Technol. Biomed. 12(1), 20 (2008)

    Article  Google Scholar 

  39. Lefebvre, G., Berlemont, S., Mamalet, F., Garcia, C.: Blstm-rnn based 3d gesture classification. In: International Conference on Artificial Neural Networks, pp. 381–388, Springer (2013

    Google Scholar 

  40. Jaimes, A., Sebe, N.: Multimodal human-computer interaction: a survey. Comput. Vis. Image Underst. 108(1–2), 116 (2007)

    Article  Google Scholar 

  41. Van Kasteren, T., Noulas, A., Englebienne, G., Kröse, B.: Accurate activity recognition in a home setting. In: Proceedings of the 10th International Conference on Ubiquitous Computing, pp. 1–9. ACM (2008)

    Google Scholar 

  42. Kwapisz, J.R., Weiss, G.M., Moore, S.A.: Activity recognition using cell phone accelerometers. ACM SigKDD Explor. Newsl. 12(2), 74 (2011)

    Article  Google Scholar 

  43. Hasan, S., Masnad, M., Mahmud, H., Hasan, M.: Human activity recognition using smartphone sensors with context filtering. In: Proceeding Ninth International Conference of Advances in Computer-Human Interactions, pp. 67–73 (2016)

    Google Scholar 

  44. Kunze, K., Lukowicz, P.: Dealing with sensor displacement in motion-based onbody activity recognition systems. In: Proceedings of the 10th International Conference on Ubiquitous Computing, pp. 20–29. ACM (2008)

    Google Scholar 

  45. Bulling, A., Roggen, D.: Recognition of visual memory recall processes using eye movement analysis. In: Proceedings of the 13th International Conference on Ubiquitous Computing, pp. 455–464. ACM (2011)

    Google Scholar 

  46. Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In Esann (2013)

    Google Scholar 

  47. Kanjo, E., Younis, E.M., Ang, C.S.: Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Inf. Fusion 49, 46 (2019)

    Article  Google Scholar 

  48. Sun, S.W., Mou, T.C., Fang, C.C., Chang, P.C., Hua, K.L., Shih, H.C.: Baseball player behavior classification system using long short-term memory with multimodal features. Sensors 19(6), 1425 (2019)

    Article  Google Scholar 

  49. Chung, S., Lim, J., Noh, K.J., Kim, G., Jeong, H.: Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning. Sensors 19(7), 1716 (2019)

    Article  Google Scholar 

  50. Neverova, N., Wolf, C., Lacey, G., Fridman, L., Chandra, D., Barbello, B., Taylor, G.: Learning human identity from motion patterns. IEEE Access 4, 1810 (2016)

    Article  Google Scholar 

  51. Yang, J., Nguyen, M.N., San, P.P., Li, X.L., Krishnaswamy, S.: Deep convolutional neural networks on multichannel time series for human activity recognition. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)

    Google Scholar 

Download references

Acknowledgements

This work is partly funded by the Irish Institute of Digital Business (dotLAB).

Author information

Authors and Affiliations

  1. Universidade de Pernambuco, Pernambuco, Brazil

    Silvano Ramos de Assis Neto, Elisson da Silva Rocha & Patricia Takako Endo

  2. Universidade Federal de Pernambuco, Pernambuco, Brazil

    Guto Leoni Santos

  3. Irish Institute of Digital Business, Dublin City University, Dublin, Ireland

    Malika Bendechache, Pierangelo Rosati & Theo Lynn

Authors
  1. Silvano Ramos de Assis Neto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guto Leoni Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elisson da Silva Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Malika Bendechache
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pierangelo Rosati
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Theo Lynn
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patricia Takako Endo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patricia Takako Endo .

Editor information

Editors and Affiliations

  1. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Hiram Ponce

  2. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Lourdes Martínez-Villaseñor

  3. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Jorge Brieva

  4. Facultad de Ingeniería, Universidad Panamericana, Mexico City, Mexico

    Ernesto Moya-Albor

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (doc 10 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

de Assis Neto, S.R. et al. (2020). Detecting Human Activities Based on a Multimodal Sensor Data Set Using a Bidirectional Long Short-Term Memory Model: A Case Study. In: Ponce, H., Martínez-Villaseñor, L., Brieva, J., Moya-Albor, E. (eds) Challenges and Trends in Multimodal Fall Detection for Healthcare. Studies in Systems, Decision and Control, vol 273. Springer, Cham. https://doi.org/10.1007/978-3-030-38748-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-38748-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-38747-1

  • Online ISBN: 978-3-030-38748-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation