Interaction with physical objects as proxy objects representing virtual objects

Patent metadata

Field	Value
Publication number	US-11069079-B2
Application number	US-201916408917-A
Country	US
Kind code	B2
Filing date	May 10, 2019
Priority date	Mar 31, 2017
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

Systems and techniques for enabling interaction with physical objects as proxy objects representing virtual objects are provided herein. Virtual reality application data associated with a virtual reality application executed on a virtual reality device, a first virtual reality object data associated with a first virtual reality object from the virtual reality application, and virtual reality event data associated with one or more events from the virtual reality application may be received. Robotic arms including a robotic hand may grasp a first physical object which corresponds to the first virtual reality object of the virtual reality application. Sensors may detect a user interaction with the first physical object. Force feedback instructions commanding the robotic arms to move while maintaining grasp of the first physical object may be generated and executed based on detecting the user interaction with the first physical object and based on the virtual reality event data.

First claim

The invention claimed is:

1. A robotic system for enabling interaction with physical objects as proxy objects representing virtual objects, comprising:
a robotic arm including a portion grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application;
a sensor detecting a user interaction with the first physical object; and
a force feedback module of a processor generating a force feedback instruction for the robotic arm, wherein the force feedback instruction commands the robotic arm and the portion to maintain grasp of the first physical object and move according to virtual reality event data, the processor executing the force feedback instruction based on detecting the user interaction with the first physical object.

2. The robotic system of claim 1 , comprising a motor control module of the processor generating a motor control release instruction for the robotic arm, wherein the motor control release instructions command the robotic arm to release grasping of the first physical object, wherein the processor executes the motor control release instruction based on the virtual reality event data.

3. The robotic system of claim 2 , wherein the motor control module of the processor generates a motor control grasp instruction for the robotic arm, wherein the motor control grasp instructions command the robotic arm to grasp a second physical object, wherein the processor executes the motor control grasp instruction based on the virtual reality event data.

4. The robotic system of claim 2 , comprising a movement portion,
wherein the sensor detects a position of a second physical object relative to the robotic system;
wherein the motor control module of the processor generates a motor control movement instruction for the movement portion;
wherein the motor control movement instructions command the movement portion to move the robotic system within a predetermined distance of the second physical object; and
wherein the processor executes the motor control movement instruction based on the virtual reality event data and a position of the user relative to the second physical object.

5. The robotic system of claim 1 , wherein the sensor detects a position of a user relative to the first physical object.

6. The robotic system of claim 1 , comprising a communication interface receiving a position of a user relative to the first physical object from a virtual reality device.

7. The robotic system of claim 6 , wherein the sensor measures an amount of force associated with the user interaction of the first physical object.

8. The robotic system of claim 7 , wherein the communication interface transmits data associated with the amount of force from the user interaction to the virtual reality device as virtual reality event data.

9. The robotic system of claim 1 , comprising:
a virtual reality event module of the processor generating a second set of virtual reality event data based on the user interaction with the first physical object, wherein the first virtual reality object is a control device within the virtual reality application; and
a communication interface transmitting the second set of virtual reality event data to a virtual reality device.

10. The robotic system of claim 1 , wherein the virtual reality event data is indicative of an event from the virtual reality application including a user approaching the first virtual reality object.

11. A method for enabling interaction with physical objects as proxy objects representing virtual objects using a robotic system, comprising:
grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application using a robotic arm including a portion;
detecting a user interaction with the first physical object;
generating a force feedback instruction for the robotic arm, wherein the force feedback instruction commands the robotic arm and the portion to maintain grasp of the first physical object and move according to virtual reality event data; and
executing the force feedback instruction based on detecting the user interaction with the first physical object.

12. The method of claim 11 , comprising:
generating a motor control release instruction for the robotic arm, wherein the motor control release instructions command the robotic arm to release grasping of the first physical object; and
executing the motor control release instruction based on the virtual reality event data.

13. The method of claim 12 , comprising:
generating a motor control grasp instruction for the robotic arm, wherein the motor control grasp instructions command the robotic arm to grasp a second physical object; and
executing the motor control grasp instruction based on the virtual reality event data.

14. The method of claim 12 , comprising:
detecting a position of a second physical object relative to the robotic system;
generating a motor control movement instruction for a movement portion to command the movement portion to move the robotic system within a predetermined distance of the second physical object; and
executing the motor control movement instruction based on the virtual reality event data and a position of the user relative to the second physical object.

15. The method of claim 11 , comprising detecting a position of a user relative to the first physical object.

16. The method of claim 11 , comprising receiving a position of a user relative to the first physical object from a virtual reality device.

17. The method of claim 11 , comprising measuring an amount of force associated with the user interaction of the first physical object.

18. The method of claim 17 , comprising transmitting data associated with the amount of force from the user interaction to the virtual reality device as virtual reality event data.

19. The method of claim 11 , wherein the virtual reality event data is indicative of an event from the virtual reality application including a user approaching the first virtual reality object.

20. A robotic system for enabling interaction with physical objects as proxy objects representing virtual objects, comprising:
a first robotic arm including a portion grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application;
a second robotic arm including a portion grasping the first physical object which corresponds to the first virtual reality object of the virtual reality application;
a sensor detecting a user interaction with the first physical object; and
a force feedback module of a processor generating force feedback instructions for each of the first robotic arm and the second robotic arm, wherein the force feedback instructions commands the first robotic arm and the second robotic arm to maintain grasp of the first physical object and move or rotate according to virtual reality event data, the processor executing the force feedback instruction based on detecting the user interaction with the first physical object.

Assignees

• Honda Motor Co Ltd

Inventors

• Murrish Robert Wesley
• Moon John P

Classifications

• B25J9/1694 Primary

  characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G05B2219/39014

  Match virtual world with real world · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title