Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

It's all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera (2310.01659v1)

Published 2 Oct 2023 in cs.CV, cs.AI, and cs.HC

Abstract: Despite significant advances in gesture recognition technology, recognizing gestures in a driving environment remains challenging due to limited and costly data and its dynamic, ever-changing nature. In this work, we propose a model-adaptation approach to personalize the training of a CNNLSTM model and improve recognition accuracy while reducing data requirements. Our approach contributes to the field of dynamic hand gesture recognition while driving by providing a more efficient and accurate method that can be customized for individual users, ultimately enhancing the safety and convenience of in-vehicle interactions, as well as driver's experience and system trust. We incorporate hardware enhancement using a time-of-flight camera and algorithmic enhancement through data augmentation, personalized adaptation, and incremental learning techniques. We evaluate the performance of our approach in terms of recognition accuracy, achieving up to 90\%, and show the effectiveness of personalized adaptation and incremental learning for a user-centered design.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. 2019. BMW’s Innovative Gesture Control Technology Sets the Industry Standard. Accessed 12-01-2023. https://news.indigoautogroup.com/bmws-innovative-gesture-control-technology-sets-the-industry-standard/
  2. 2021. The new Mercedes-Maybach S-Class up close: MBUX Interior Assist Rear. Accessed 12-01-2023. https://group-media.mercedes-benz.com/marsMediaSite/en/instance/ko/The-new-Mercedes-Maybach-S-Class-up-close-MBUX-Interior-Assist-Rear.xhtml?oid=50185650
  3. You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing. In Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 595–603. https://doi.org/10.1145/3382507.3418836
  4. Selection facilitation schemes for predictive touch with mid-air pointing gestures in automotive displays. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 21–32.
  5. Stephen R Anderson. 2010. How many languages are there in the world. Linguistic Society of America (2010). 6 pages.
  6. Joanna J. Bryson and Andreas Theodorou. 2019. How Society Can Maintain Human-Centric Artificial Intelligence. Springer Singapore, Singapore, 305–323.
  7. Gesture and Voice Commands to Interact With AR Windshield Display in Automated Vehicle: A Remote Elicitation Study. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Seoul, Republic of Korea) (AutomotiveUI ’22). Association for Computing Machinery, New York, NY, USA, 171–182. https://doi.org/10.1145/3543174.3545257
  8. Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and vision computing 21, 8 (2003), 745–758.
  9. An improved hand gesture recognition system using keypoints and hand bounding boxes. Array (2022), 100251. https://doi.org/10.1016/j.array.2022.100251
  10. A survey of glove-based systems and their applications. IEEE transactions on systems, man, and cybernetics, part c (applications and reviews) 38, 4 (2008), 461–482.
  11. Hidden markov model based continuous online gesture recognition. In Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No. 98EX170), Vol. 2. IEEE, 1206–1208.
  12. Designing a user-defined gesture vocabulary for an in-vehicle climate control system. In Proceedings of the 28th Australian Computer-Human Interaction Conference. ACM, 391–395.
  13. Driver queries using wheel-constrained finger pointing and 3-D head-up display visual feedback. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 56–62.
  14. Alexander Gepperth and Barbara Hammer. 2016. Incremental learning algorithms and applications. In European symposium on artificial neural networks (ESANN).
  15. Amr Gomaa. 2022. Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces. In Proceedings of the 2022 International Conference on Multimodal Interaction (Bengaluru, India) (ICMI ’22). Association for Computing Machinery, New York, NY, USA, 690–695. https://doi.org/10.1145/3536221.3557034
  16. Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle. In Proceedings of the 2020 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 501–509. https://doi.org/10.1145/3382507.3418817
  17. ML-PersRef: A Machine Learning-Based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle. In Proceedings of the 2021 International Conference on Multimodal Interaction. Association for Computing Machinery, New York, NY, USA, 318–327. https://doi.org/10.1145/3462244.3479910
  18. Hand gesture recognition for human computer interaction. Procedia Computer Science 115 (2017), 367–374.
  19. Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Systems with Applications 38, 5 (2011), 6031–6042.
  20. Pan Jing and Guan Ye-Peng. 2013. Human-computer interaction using pointing gesture based on an adaptive virtual touch screen. International Journal of Signal Processing, Image Processing and Pattern Recognition 6, 4 (2013), 81–91.
  21. Roland Kehl and Luc Van Gool. 2004. Real-time pointing gesture recognition for an immersive environment. In Proceedings of the 6th International Conference on Automatic Face and Gesture Recognition. IEEE, 577–582.
  22. Real-time hand gesture detection and classification using convolutional neural networks. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 1–8.
  23. Hand Gesture and Human-Drone Interaction. In Intelligent Systems and Applications, Kohei Arai (Ed.). Springer International Publishing, Cham, 299–308.
  24. Deep learning. nature 521, 7553 (2015), 436–444.
  25. Hand gesture recognition based on convolution neural network. Cluster Computing 22, 2 (2019), 2719–2729.
  26. Human hand gesture recognition using a convolution neural network. In 2014 IEEE International Conference on Automation Science and Engineering (CASE). IEEE, 1038–1043.
  27. A Two-Stage Convolutional Neural Network for Hand Gesture Recognition. In Proceedings of the 6th International Conference on Advance Computing and Intelligent Engineering, Bibudhendu Pati, Chhabi Rani Panigrahi, Prasant Mohapatra, and Kuan-Ching Li (Eds.). Springer Nature Singapore, Singapore, 383–392.
  28. Manar Maraqa and Raed Abu-Zaiter. 2008. Recognition of Arabic Sign Language (ArSL) using recurrent neural networks. In 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT). IEEE, 478–481.
  29. Tin Hninn Hninn Maung. 2009. Real-time hand tracking and gesture recognition system using neural networks. International Journal of Computer and Information Engineering 3, 2 (2009), 315–319.
  30. Hand gesture recognition using hidden Markov models. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, Vol. 5. IEEE, 4232–4235.
  31. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE, 1–7.
  32. Multi-sensor system for driver’s hand-gesture recognition. In Proceedings of the 11th International Conference on Automatic Face and Gesture Recognition. IEEE, 1–8.
  33. Mohammad Mehdi Moniri and Christian Müller. 2012. Multimodal reference resolution for mobile spatial interaction in urban environments. In Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 241–248.
  34. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In 2011 IEEE international conference on signal and image processing applications (ICSIPA). IEEE, 342–347.
  35. Combining speech, gaze, and micro-gestures for the multimodal control of in-car functions. In Proceedings of the 12th International Conference on Intelligent Environments. IEEE, 190–193.
  36. A multi-scale approach to gesture detection and recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 484–491.
  37. 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction scenario. In Proceedings of the 6th International Conference on Automatic Face and Gesture Recognition. IEEE, 565–570.
  38. Assessing artificial intelligence for humanity: Will AI be the our biggest ever advance? Or the biggest threat [Opinion]. IEEE Technology and Society Magazine 37, 4 (2018), 26–34.
  39. Eshed Ohn-Bar and Mohan Manubhai Trivedi. 2014. Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations. IEEE Transactions on Intelligent Transportation Systems 15, 6 (2014), 2368–2377.
  40. A Human-Robot Interaction System Based on Calling Hand Gestures. In Intelligent Computing Methodologies, De-Shuang Huang, Zhi-Kai Huang, and Abir Hussain (Eds.). Springer International Publishing, Cham, 43–52.
  41. A research study of hand gesture recognition technologies and applications for human vehicle interaction. In Proceedings of the 3rd Institution of Engineering and Technology conference on automotive electronics. IET, 1–15.
  42. Static hand gesture recognition based on convolutional neural networks. Journal of Electrical and Computer Engineering 2019 (2019).
  43. The design of hand gestures for human-computer interaction: Lessons from sign language interpreters. International Journal of Human Computer Studies 72, 10-11 (10 2014), 728–735.
  44. Yu Ren and Fengming Zhang. 2009. Hand gesture recognition based on MEB-SVM. In 2009 International Conference on Embedded Software and Systems. IEEE, 344–349.
  45. High performance real-time gesture recognition using hidden markov models. In International Gesture Workshop. Springer, 69–80.
  46. Florian Roider and Tom Gross. 2018. I see your point: Integrating gaze to enhance pointing gesture accuracy while driving. In Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 351–358.
  47. The effects of situational demands on gaze, speech and gesture input in the vehicle. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 94–102.
  48. Free-hand pointing for identification and interaction with distant objects. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 40–47.
  49. Convolutional, long short-term memory, fully connected deep neural networks. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 4580–4584.
  50. Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504.
  51. Dushyant Kumar Singh. 2015. Recognizing hand gestures for human computer interaction. In Proceedings of the International Conference on Communications and Signal Processing. IEEE, 379–382.
  52. Ekaterini Stergiopoulou and Nikos Papamarkos. 2009. Hand gesture recognition using a neural network shape fitting technique. Engineering Applications of Artificial Intelligence 22, 8 (2009), 1141–1158.
  53. User Experience of In-Vehicle Gesture Interaction: Exploring the Effect of Autonomy and Competence in a Mock-Up Experiment. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Seoul, Republic of Korea) (AutomotiveUI ’22). Association for Computing Machinery, New York, NY, USA, 285–296. https://doi.org/10.1145/3543174.3546847
  54. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1–9.
  55. Tomoichi Takahashi and Fumio Kishino. 1991. Hand Gesture Coding Based on Experiments Using a Hand Gesture Interface Device. SIGCHI Bull. 23, 2 (mar 1991), 67–74. https://doi.org/10.1145/122488.122499
  56. A neural network model for development of reaching and pointing based on the interaction of forward and inverse transformations. Developmental science 21, 3 (2018). 10 pages.
  57. Gesture Recognition with a Convolutional Long Short-Term Memory Recurrent Neural Network.. In ESANN.
  58. Wei Xu. 2019. Toward human-centered AI: a perspective from human-computer interaction. interactions 26, 4 (2019), 42–46.
  59. Hand-free gesture recognition for vehicle infotainment system control. In Proceedings of the IEEE Vehicular Networking Conference. IEEE, 1–2.
  60. Implementation and evaluation of touch and gesture interaction modalities for in-vehicle infotainment systems. In Image and Graphics, Yao Zhao, Nick Barnes, Baoquan Chen, Rüdiger Westermann, Xiangwei Kong, and Chunyu Lin (Eds.). Springer, 384–394.
  61. A Hand Gesture Interface Device. In Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface (Toronto, Ontario, Canada) (CHI ’87). Association for Computing Machinery, New York, NY, USA, 189–192. https://doi.org/10.1145/29933.275628
  62. A real-time system for hand gesture controlled operation of in-car devices. In Proceedings of the International Conference on Multimedia and Expo (ICME). IEEE. 4 pages.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now