Mobile terminal for controlling VR image and control method therefor

Patent metadata

Field	Value
Publication number	US-11308704-B2
Application number	US-201616071072-A
Country	US
Kind code	B2
Filing date	Jan 18, 2016
Priority date	Jan 18, 2016
Publication date	Apr 19, 2022
Grant date	Apr 19, 2022

Abstract

The present invention relates to a mobile terminal and a control method therefor, the mobile terminal comprising: at least one camera; a display unit placed toward the eyes of a user searing the mobile terminal and for displaying an image, the image comprising one of a VR (virtual reality) image and a 3D image; and a control unit for performing an action related to the image according to a gesture input sensed by the camera, wherein, when a first gesture input proceeding along a first plane opposite to the image is sensed or a second gesture input proceeding along a second plane orthogonal to the first plane is sensed by the camera, the control unit performs an action corresponding to the sensed first or second gesture input, thereby providing the user with an effect of providing more various VR related functions through the first gesture input made on a first plane space opposite to the image visible to the eyes of the user and the second gesture input made on a second plane space orthogonal to the first plane.

First claim

What is claimed is:

1. A mobile terminal comprising:
at least a first artificial intelligence (AI) camera;
a display unit disposed to be viewable by a user wearing the mobile terminal, the display unit configured to display a virtual reality (VR) image extending along a first plane; and
a processor configured to:
perform an action related to the VR image in response to a VR gesture input detected through the first AI camera,
wherein, based on the detected VR gesture input comprising a VR gesture input moving along a second plane parallel to the first plane, the processor is further configured to control the display unit to display a cursor having a shape of the VR gesture input moving along the second plane, and
wherein, based on the detected VR gesture input comprising a VR gesture input moving along a third plane orthogonal to the first plane, the processor is further configured to control the display unit to display a cursor having a shape of the VR gesture input moving along the third plane.

2. The mobile terminal of claim 1 , wherein the processor is further configured to:
recognize the detected VR gesture input as being the VR gesture input moving along the second plane based on the detected VR gesture input changing a distance along the second plane between a first pointer and a second pointer when the first pointer and the second pointer are located in the second plane, and
recognize the detected VR gesture input as being the VR gesture input moving along the third plane based on the detected VR gesture input changing a distance along the third plane between the first pointer and the second pointer when the first pointer and the second pointer are located in the third plane.

3. The mobile terminal of claim 2 , wherein the processor is further configured to perform a different action according to a magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer in response to the detected VR gesture input.

4. The mobile terminal of claim 3 , wherein the processor is further configured to:
when the detected VR gesture input is input at a position of a specific object included in the VR image, control the display unit to enlarge or reduce the specific object in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer, and
when the detected VR gesture input is input at a position different from the position of the specific object, control the display unit to enlarge or reduce the whole VR image in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer.

5. The mobile terminal of claim 3 , wherein the processor is further configured to:
when the gesture input moving along the second plane is input at a position of a specific object included in the VR image, control the display unit to enlarge the specific object; and
when the gesture input moving along the third plane is input at the position of the specific object, control the display unit to reduce the specific object.

6. The mobile terminal of claim 3 , wherein the processor is further configured to:
when the detected VR gesture input is input at a position of a 3D object in the VR image, control the display unit to change a 3D depth level of the 3D object in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer; and
when the detected VR gesture input is input at a position different from the position of the 3D object in the VR image, control the display unit to change a 3D depth level of the VR image in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer.

7. The mobile terminal of claim 3 , wherein the processor is further configured to:
when the detected VR gesture input is input at a position of a 3D object in the VR image, control the display unit to rotate the 3D object in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer; and
when the detected VR gesture input is input at a position different from the position of the 3D object in the VR image, control the display unit to rotate the whole VR image in response to the magnitude of the change in the distance along the second plane or the third plane between the first pointer and the second pointer.

8. The mobile terminal of claim 2 , wherein the processor is further configured to:
while the first pointer and the second pointer are located at a position of a specific object included in the VR image, change the specific object into a 3D object and then control the display unit to display the 3D object, when the first pointer and the second pointer are rotated in a direction from the second plane to the third plane or a direction from the third plane to the second plane; and
while the first pointer and the second pointer are located at a position different from the position of the specific object in the VR image, change the whole VR image into a 3D image and then control the display unit to display the 3D image, when the first pointer and the second pointer are rotated in the direction from the second plane to the third plane or the direction from the third plane to the second plane.

9. The mobile terminal of claim 1 , wherein the processor is further configured to:
when the VR gesture input moving along the second plane or the VR gesture input moving along the third plane is input at a position of at least a first object included in the VR image, control the display unit to list and display sub-objects subordinate to the first object; and
when the VR gesture input moving along the second plane or the VR gesture input moving along the third plane is input at a position different from the position of the at least the first object in the VR image, control the display unit to list and display all objects included in the VR image.

10. A method of controlling a mobile terminal, the method comprising:
displaying a virtual reality (VR) image extending along a first plane;
detecting, by a first artificial intelligence (AI) camera, a VR gesture input moving along a second plane parallel to the first plane;
based on the detected VR gesture input moving along the second plane, displaying a cursor having a shape of the VR gesture input moving along the second plane;
detecting, by the first AI camera, a VR gesture input moving along a third plane orthogonal to the first plane; and
based on the detected VR gesture input moving along the third plane, displaying a cursor having a shape of the VR gesture input moving along the third plane.

11. The method of claim 10 , wherein:
detecting the VR gesture input moving along the second plane comprises recognizing that the VR gesture input moving along the second plane changes a distance along the second plane between a first pointer and a second pointer when the first pointer and the second pointer are located in the second plane, and
detecting the VR gesture input moving along the third plane comprises recognizing that the VR gesture input moving along the third plane changes a distance along the third plane between the first pointer and the second pointer when the first pointer and the second pointer are located in the third plane.

12. The method of claim 11 , further comprising:
performing a different action according to a magnitude of the change in the distance along the second plane o

Assignees

• Lg Electronics Inc

Inventors

• Ryu Eunjeong
• Kim Youngjun
• Park Wonjoo

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2207/30242

  Counting objects in image · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title