Person-in-WiFi 3D: End-to-End Multi-Person 3D Pose Estimation with Wi-Fi

Abstract

Wi-Fi signals, in contrast to cameras, offer privacy protection and occlusion resilience for some practical scenarios such as smart homes, elderly care, and virtual reality. Recent years have seen remarkable progress in the estimation of single-person 2D pose, single-person 3D pose, and multi-person 2D pose. This paper takes a step forward by introducing Person-in-WiFi 3D, a pioneering Wi-Fi system that accomplishes multi-person 3D pose estimation. Person-in-WiFi 3D has two main updates. Firstly, it has a greater number of Wi-Fi devices to enhance the capability for capturing spatial reflections from multiple individuals. Secondly, it leverages the Transformer for end-to-end estimation. Compared to its predecessor, Person-in-WiFi 3D is storage-efficient and fast. We deployed a proof-of-concept system in 4m * 3.5m areas and collected a dataset of over 97K frames with seven volunteers. Person-in-WiFi 3D attains 3D joint localization errors of 91.7mm (1-person), 108.1mm (2-person), and 125.3mm (3-person), comparable to cameras and millimeter-wave radars.

Visualization results

Visualization of pose estimation in three different scenarios

---

---

Wi-Fi Mesh

We could simply modify the prediction head of Person-in-WiFi 3D to expanding it from multi-person 3D pose estimation to multi-person 3D mesh reconstruction. Here are some visualization results.

Mesh reconstruction visualization results

Visualization of 3D human Mesh in three different scenarios

---

---

Citation

@inproceedings{person3dyan,
  title={Person-in-WiFi 3D: End-to-End Multi-Person 3D Pose Estimation with Wi-Fi },
  author={Yan, Kangwei and Wang, Fei and Qian, Bo and Ding, Han and Han, Jinsong and Wei, Xing},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}

Some our previous work

• Fei Wang, YiZhe Lv, Mengdie Zhu, Han Ding, Jinsong Han. XRF55: A Radio Frequency Dataset for Human Indoor Action Analysis. ACM IMWUT/Ubicomp 2024 , Project site


• Fei Wang, Sanping Zhou, Stanislav Panev, Jinsong Han, Dong Huang. Person-in-WiFi: Fine-grained person perception using WiFi. IEEE/CVF ICCV 2019.


• Fei Wang, Jinsong Han, Feng Lin, Kui Ren. Wipin: Operation-free passive person identification using wi-fi signals. IEEE GLOBECOM 2019 (Best Paper Award).


• Dahu Shi, Xing Wei, Liangqi Li, Ye Ren, Wenming Tan. End-to-end multi-person pose estimation with transformers. IEEE/CVF CVPR 2022

Acknowledgements

We are grateful to anonymous reviewers for the invaluable comments. We thank all volunteers for their participations.
The website template is taken from Custom Diffusion (which was built on DreamFusion's project page). The text editor used in the demo video has been taken from Rich Text-to-Image.

Person-in-WiFi-3d by Kangwei Yan,Fei Wang, Bo Qian, Han Ding, Jinsong Han, Xing Wei is licensed under Attribution-NonCommercial 4.0 International