Remote driving with a virtual reality system

Patent metadata

Field	Value
Publication number	US-9568995-B1
Application number	US-201514983620-A
Country	US
Kind code	B1
Filing date	Dec 30, 2015
Priority date	Dec 30, 2015
Publication date	Feb 14, 2017
Grant date	Feb 14, 2017

Abstract

A method and system for facilitating a user drive an automobile remotely in real-time through a virtual reality system is disclosed. The virtual reality environment may present real-time road conditions, weather condition, automobile information, and/or any other information regarding driving of the automobile in real time.

First claim

What is claimed is:

1. A computer-implemented method for facilitating a user to remotely drive an automobile through a virtual reality system, the method comprising:
receiving, by a server, from an automobile, a first geo-location of the automobile at a first time instance;
based on the first geo-location received, rendering, by the server, a first view of a virtual reality environment reflecting a road the automobile travels in at the first time instance, wherein the rendering of the first view of the virtual reality environment includes:
obtaining, from a scene file database, pre-rendered scene files corresponding to the first geo-location, the scene files representing an environment of the road at the first geo-location;
obtaining, from a real-time database, condition information regarding one or more real-time conditions about the road at the first geo-location; and
constructing the first view of the virtual reality environment using the pre-rendered scene files and the condition information;

transmitting the first view of the virtual reality environment to a display device associated with the user;
receiving, by the server, a first control input from a first interactive control device associated with the user;
based on the first control input, determining, by the server, a second view of the virtual reality environment; and
rendering, by the server, one or more images representing the second view.

2. The computer-implemented method of claim 1 , wherein the first control input includes information indicating a translational or a rotational movement of the first interactive control device.

3. The computer-implemented method of claim 1 , wherein the first interactive control device is a virtual steering wheel, a physical steering wheel, a physical brake pad, or a physical gas pedal.

4. The computer-implemented method of claim 1 , further comprising receiving view information regarding a first view point of the first user into the environment of the road at the first geo-location; and wherein, the rendering of the first view of the virtual reality environment is further based on the view point information.

5. The computer-implemented method of claim 1 , wherein the determination of the second view includes determining a change of the first view based on the first control input.

6. The computer-implemented method of claim 1 , wherein the determination of the second view includes determining driving information regarding the automobile at the time instance.

7. The computer-implemented method of claim 1 , wherein rendering the second view includes determining driving information regarding the automobile for presentation on a virtual instrument panel in the VR environment.

8. The computer-implemented method of claim 1 , further comprising receiving, by the server, from a head mount device (HMD) neck movement information indicating a neck movement of a user that remotely drives the automobile, and wherein the second view is determined further based on the neck movement.

9. The computer-implemented method of claim 1 , further comprising
receiving, by the server, from a second automobile, a second geo-location of the second automobile at a first time instance;
based on the second geo-location received, rendering, by the server, a third view of a virtual reality environment reflecting a road the second automobile travels in at the first time instance;
receiving, by the server, a second control input from a first interactive control device associated with the automobile;
based on the first control input, determining, by the server, a fourth view of the virtual reality environment; and
rendering one or more images representing the second view.

10. A virtual reality system for facilitating a user to remotely drive an automobile, the virtual reality system comprising a processor configured to execute machine-readable instruction, when executed, causing the processor to perform:
receiving, from an automobile, a first geo-location of the automobile at a first time instance;
based on the first geo-location received, rendering, by the server, a first view of a virtual reality environment reflecting a road the automobile travels in at the first time instance, wherein the rendering of the first view of the virtual reality environment includes:
obtaining, from a scene file database, pre-rendered scene files corresponding to the first geo-location, the scene files representing an environment of the road at the first geo-location;
obtaining, from a real-time database, condition information regarding one or more real-time conditions about the road at the first geo-location; and
constructing the first view of the virtual reality environment using the pre-rendered scene files and the condition information;

transmitting the first view of the virtual reality environment to a display device associated with the user;
receiving, by the server, a first control input from a first interactive control device associated with the user;
based on the first control input, determining, by the server, a second view of the virtual reality environment; and
rendering one or more images representing the fourth view.

11. The system of claim 10 , wherein the first control input includes information indicating a translational or a rotational movement of the first interactive control device.

12. The system of claim 10 , wherein the first interactive control device is a virtual steering wheel, a physical steering wheel, a physical brake pad, or a physical gas pedal.

13. The system of claim 10 , wherein the machine-readable instructions are further configured to cause the processor to perform receiving view information regarding a first view point of the first user into the environment of the road at the first geo-location; and wherein, the rendering of the first view of the virtual reality environment is further based on the view point information.

14. The system of claim 10 , wherein the determination of the second view includes determining a change of the first view based on the first control input.

15. The system of claim 10 , wherein the determination of the second view includes determining driving information regarding the automobile at the time instance.

16. The system of claim 10 , wherein rendering the second view includes determining driving information regarding the automobile for presentation on a virtual instrument panel in the VR environment.

17. The system of claim 10 , wherein the machine-readable instructions further causing the processor to perform receiving from a head mount device (HMD) neck movement information indicating a neck movement of a user that remotely drives the automobile, and wherein the second view is determined further based on the neck movement of the user.

18. The system of claim 10 , wherein the machine-readable instructions further cause the processor to perform
receiving, by the server, from a second automobile, a second geo-location of the second automobile at a first time instance;
based on the second geo-location received, rendering, by the server, a third view of a virtual reality environment reflecting a road the second automobile travels in at the first time instance;
receiving, by the server, a second control input from a first interactive control device associated with the automobile;
based on the first control input, determining, by the server, a fourth view of the virtual reality environment; and
rendering one or more images representing the fourth view.

Assignees

• Thunder Power Hong Kong Ltd

Inventors

• Lian Po-Han

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2210/61

  Scene description · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G05D1/0044

  by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005) · CPC title