Human-Robot Interaction

VR and Digital Twin Based Teleoperation of Robotic Arm

This research proposes a novel telemanipulation system that enables a human operator to control a dual-arm robot. The operation provides kinesthetic teaching via a digital twin of the robot which the operator cyber-physically guides to perform a task. Its key enabler is the concept of a virtual reality interactive marker, which serves as a simplified end effector of the digital twin robot. In virtual reality, the operator can interact with the marker using bare hands, which are sensed by the Leap Motion on top of a virtual reality headset. Then, the status (e.g. position/orientation) of the marker is transformed to the corresponding joint space command to the remote robot so that its end effector can follow the marker.

Key Publications

• Intuitive Bare-Hand Teleoperation of a Robotic Manipulator using Virtual Reality and Leap Motion