HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand

OVERVIEW

HandProxy is a speech-driven system that lets users perform rich, hands-on interactions in virtual environments by controlling a virtual proxy hand with natural language.

• Converts spoken commands into real-time, nuanced control of virtual hands for tasks like UI navigation, object manipulation, and gesture-based actions.
• Unlocks new opportunities for hands-free accessibility and natural interaction in XR, targeting gaming, productivity, healthcare, and inclusive technology.

BACKGROUND

Hand gestures power expressive, intuitive interaction in XR applications (virtual and augmented reality), but they’re not always feasible. Environmental constraints, physical barriers, or user motor limitations can exclude people from these experiences. While many XR devices offer speech control, today’s systems are narrowly limited—supporting just a handful of rigid commands and unable to reproduce the depth of real hand gestures.

There’s strong market demand for XR technologies that expand accessibility and adapt to varied user abilities—particularly as immersive content grows in entertainment, work, and training. Trends show increasing consumer expectations (and regulation) around inclusion, hands-free control, and flexible, natural ways to interface with virtual systems.

INNOVATION

HandProxy’s technical breakthrough is interpreting everyday speech, whether high-level requests ("maximize the volume," "grab the left fruit and move it up") or step-by-step instructions, and seamlessly animating a virtual hand to execute those instructions in XR environments. Its AI parses speech in context, breaks commands into actionable primitives (gesture, target, spatial and temporal controls), and delivers responsive interaction with visual feedback for clarity and error recovery.

This approach enables:

• Expressive, multi-step hand interactions from natural speech—no memorization of commands required.
• Reliable disambiguation using environmental context and AI reasoning, so users don’t get stuck on phrasing or specifics.
• Visual overlays for command previews and corrections, making interaction transparent and easy to refine.
• Proven usability, with a user study finding 100% task completion and high execution accuracy using diverse spoken input.

HandProxy makes immersive environments truly hands-free and accessible, offering the expressive flexibility of hand gestures while removing the barriers of physical controls.

ADDITIONAL INFORMATION

REFERENCES:

"HandProxy: Expanding the Affordances of Speech Interfaces in Immersive Environments with a Virtual Proxy Hand"

INTELLECTUAL PROPERTY:

Patent application pending