Improving Real-Time Omnidirectional 3D Multi-Person Human Pose Estimation with People Matching and Unsupervised 2D-3D Lifting (2403.09437v1)
Abstract: Current human pose estimation systems focus on retrieving an accurate 3D global estimate of a single person. Therefore, this paper presents one of the first 3D multi-person human pose estimation systems that is able to work in real-time and is also able to handle basic forms of occlusion. First, we adjust an off-the-shelf 2D detector and an unsupervised 2D-3D lifting model for use with a 360$\circ$ panoramic camera and mmWave radar sensors. We then introduce several contributions, including camera and radar calibrations, and the improved matching of people within the image and radar space. The system addresses both the depth and scale ambiguity problems by employing a lightweight 2D-3D pose lifting algorithm that is able to work in real-time while exhibiting accurate performance in both indoor and outdoor environments which offers both an affordable and scalable solution. Notably, our system's time complexity remains nearly constant irrespective of the number of detected individuals, achieving a frame rate of approximately 7-8 fps on a laptop with a commercial-grade GPU.
- K. Ludwig, S. Scherer, M. Einfalt, and R. Lienhart, “Self-supervised learning for human pose estimation in sports,” in 2021 IEEE International Conference On Multimedia & Expo Workshops (ICMEW), pp. 1–6, 2021.
- L. Kumarapu and P. Mukherjee, “AnimePose: Multi-person 3D pose estimation and animation,” Pattern Recognition Letters, vol. 147, pp. 16–24, 2021.
- M. Martin, S. Stuehmer, M. Voit, and R. Stiefelhagen, “Real time driver body pose estimation for novel assistance systems,” in 2017 IEEE 20th International Conference On Intelligent Transportation Systems (ITSC), pp. 1–7, 2017.
- M. Furst, S. Gupta, R. Schuster, O. Wasenmuller, and D. Stricker, “HPERL: 3D Human Pose Estimation from RGB and LiDAR,” 2020.
- C. Keskin and et al., “Real time hand pose estimation using depth sensors,” in Consumer Depth Cameras for Computer Vision, pp. 119–137, 2013.
- Z. Cao, G. Hidalgo Martinez, T. Simon, S.-E. Wei, and Y. Sheikh, “Openpose: Realtime multi-person 2d pose estimation using part affinity fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2019.
- P. Hardy and H. Kim, “Links ”lifting independent keypoints” – partial pose lifting for occlusion handling with improved accuracy in 2d-3d human pose estimation,” 2023.
- A. Aarti, T. Alberto, K. Isaac, S. Emil, F. Timothy, L. Hwasup, and K. Hansung, “Real-time 3d multi-person pose estimation using an omnidirectional camera and mmwave radars,” in Proc. ICEET, October 2023.
- Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on pattern analysis and machine intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.
- J. Oh, K.-S. Kim, M. Park, and S. Kim, “A comparative study on camera-radar calibration methods,” in 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 1057–1062, 2018.
- K. Garcia, “Bringing intelligent autonomy to fine motion detection and people counting with ti mmwave sensors,” 2019.
- C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, “Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, pp. 1325–1339, jul 2014.