Real-time and adaptive social gesture perception for human-robot interaction.
assets/demo.mp4
Robots interacting with people must recognize not only explicit commands, but also social cues such as invitations, refusals, and unavailability. In real deployments these cues have to be inferred from noisy onboard perception, under partial occlusion, shifting viewpoints and a hard latency budget.
SocioGesture pairs a compact confidence-aware body-hand skeleton representation with a lightweight dual-stream model that fuses body motion and hand articulation in a single forward pass. Training on deliberately corrupted skeletons — missing hands, occluded arms, keypoints that flicker in and out — buys robustness without costing anything at inference time. On a social gesture dataset collected in mixed indoor and outdoor settings, the system generalizes to held-out subjects, holds up under structured joint occlusion, and runs in real time on a robot-mounted edge device.
During deployment the robot saves the interaction segments it was unsure about. Those clips are labeled offline and folded back in, which lets the gesture vocabulary grow without eroding the classes the robot already handled.
Every gesture resolves to one conservative action. Above the confidence threshold the robot commits; below it, it holds position and files the clip away instead of guessing.
A 30 FPS camera leaves 33 ms per frame. The full one-person stack spends 25 ms of it on the robot's Jetson Thor, so recognition keeps pace with the video rather than lagging behind it.
The recognizer itself is 1.3 M parameters and 2.3 GFLOPs, evaluated in one forward pass rather than an ensemble of joint, bone and motion streams.
Hands and arms carry most of the signal and are exactly what the pose estimator loses first.
| Training | Hands hidden | Arm hidden |
|---|---|---|
| Standard | 31.8 | 28.3 |
| Occlusion-aware | 84.9 | 79.5 |
Uncertain clips from deployment were labeled offline, verified, then folded back in.
| Clip accuracy | Before | After |
|---|---|---|
| All 10 gestures | 64.6 | 87.9 |
| Original 7 | 98.5 | 98.5 |
| Added 3 | — | 67.6 |
Live, on the G1 with five people who appear nowhere in the training, fine-tuning or offline test data: 146 of 150 trials recognized correctly. When the robot did act, it chose the right behavior 98.3% of the time, and no stop gesture ever produced an approach. The failures are almost all the robot declining to move, which is the direction we want it to fail in.
We're preparing the dataset and evaluation code for release and will link them here once they're ready. If you'd like early access for a specific comparison, get in touch — wendy@openmind.com.
Update the eprint number and year once the arXiv identifier is assigned.
@article{fu2026sociogesture,
title = {SocioGesture: Real-Time and Adaptive Social Gesture
Perception for Human-Robot Interaction},
author = {Fu, Wenjin and Wu, Li-Fan and Peter, Jerin and
Huyen, Chip and Chen, Boyuan and Liphardt, Jan},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}