Wearable electronic device for displaying virtual object and method of controlling the same

Patent metadata

Field	Value
Publication number	US-2023135420-A1
Application number	US-202217979306-A
Country	US
Kind code	A1
Filing date	Nov 2, 2022
Priority date	Nov 2, 2021
Publication date	May 4, 2023
Grant date	—

Abstract

A wearable electronic device may include a display, a camera, at least one first sensor, and at least one processor operatively coupled to the display, the camera, and the at least one first sensor. The at least one processor may be configured to display an execution screen of an application on the display, based on receiving a first user input for selecting a content from among a plurality of contents included in the execution screen of the application through the camera, display a first virtual object related to the selected content on the display, based on receiving a second user input for mapping the first virtual object to an object included in a real space through the camera, identify a type of the object in the real space through the at least one first sensor, in response to the type of the object in the real space being related to the selected content or the first virtual object, display on the display a second virtual object related to the selected content based on at least one of a shape, a position, or a size of the object in the real space, and in response to the type of the object in the real space being not related to the selected content or the first virtual object, display on the display a third virtual object in the form of a three-dimensional (3D) object related to the selected content.

First claim

1 . A wearable electronic device comprising:
a display; a camera; at least one first sensor; and at least one processor operatively coupled to the display, the camera, and the at least one first sensor, wherein the at least one processor is configured to: control to display an execution screen of an application on the display, based at least on receiving a first user input for selecting a content from among a plurality of contents included in the execution screen of the application through the camera, control to display a first virtual object related to the selected content on the display, based at least on receiving a second user input for mapping the first virtual object to an object included in a real space through the camera, identify a type of the object in the real space through the at least one first sensor, and in response to at least the type of the object in the real space being related to the selected content and/or the first virtual object, control to display on the display a second virtual object related to the selected content based at least on at least one of a shape, a position, or a size of the object in the real space, and in response to at least the type of the object in the real space being not related to the selected content and/or the first virtual object, control to display on the display a third virtual object in the form of a three-dimensional (3D) object related to the selected content.

2 . The wearable electronic device of claim 1 , wherein the at least one processor is configured to determine a shape of the third object based at least on the type of the object in the real space.

3 . The wearable electronic device of claim 1 , further comprising a memory,
wherein the at least one processor is configured to identify the type of the object in the real space based at least on an artificial intelligence model stored in the memory.

4 . The wearable electronic device of claim 1 , wherein the at least one processor is configured to, in response to at least an event related to the selected content being occurred, update the second virtual object and/or the third virtual object based at least on information about the event.

5 . The wearable electronic device of claim 4 , wherein the at least one processor is configured to change a shape of the second virtual object and/or a shape of the third virtual object based at least on the information about the event.

6 . The wearable electronic device of claim 1 , further comprising at least one second sensor,
wherein the at least one processor is configured to: detect a gaze of a user through the at least one second sensor, and based at least on the detected gaze of the user being directed toward the object in the real space, display the second virtual object and/or the third virtual object on the display, and based at least on the detected gaze of the user not being directed toward the object in the real space, delete part of the second virtual object and/or part of the third virtual object.

7 . The wearable electronic device of claim 1 , wherein the second virtual object includes content information displayed in at least one of at least part of the object in the real space or the vicinity of the object in the real space.

8 . The wearable electronic device of claim 1 , wherein the at least one processor is configured to, based at least on receiving a third user input for selecting the second virtual object and/or the third virtual object, display the execution screen of the application related to the selected content around the second virtual object and/or the third virtual object.

9 . The wearable electronic device of claim 1 , wherein the at least one processor is configured to:
based at least on receiving the first user input, identify a plurality of objects included in the real space, control to display a plurality of icons indicating that the first virtual object is mappable, respectively on the plurality of objects, and based at least on receiving the second user input on one of the plurality of icons, identify a type of an object in the real space, corresponding to the one icon through the at least one first sensor.

10 . The wearable electronic device of claim 1 , further comprising a communication module comprising communication circuitry,
wherein the at least one processor is configured to: based at least on identification of occurrence of an event related to the second virtual object and/or the third virtual object through the camera, identify whether the wearable electronic device is connected to an external electronic device through the communication module, and based at least on identification that the wearable electronic device is connected to the external electronic device, control to transmit information about the event to the external electronic device through the communication module.

11 . A method of controlling a wearable electronic device, the method comprising:
displaying an execution screen of an application on a display of the wearable electronic device; based at least on receiving a first user input for selecting a content from among a plurality of contents included in the execution screen of the application through a camera of the wearable electronic device, displaying a first virtual object related to the selected content on the display; based at least on receiving a second user input for mapping the first virtual object to an object included in a real space through the camera, identifying a type of the object in the real space through at least one first sensor of the wearable electronic device; and in response to at least the type of the object in the real space being related to the selected content and/or the first virtual object, displaying on the display a second virtual object related to the selected content based on at least one of a shape, a position, or a size of the object in the real space, and in response to at least the type of the object in the real space being not related to the selected content or the first virtual object, displaying on the display a third virtual object in the form of a three-dimensional (3D) object related to the selected content.

12 . The method of claim 11 , further comprising determining a shape of the third object based on the type of the object in the real space.

13 . The method of claim 11 , wherein identifying the type of the object comprises identifying the type of the object in the real space based on an artificial intelligence model stored in memory of the wearable electronic device.

14 . The method of claim 11 , further comprising, in response to an event related to the selected content being occurred, updating the second virtual object or the third virtual object based on information about the event.

15 . The method of claim 14 , further comprising changing a shape of the second virtual object and/or a shape of the third virtual object based at least on the information about the event.

16 . The method of claim 11 , further comprising:
detecting a gaze of a user through at least one second sensor of the wearable electronic device; and based at least on the detected gaze of the user being directed toward the object in the real space, displaying the second virtual object and/or the third virtual object on the display, and based on the detected gaze of the user not being directed toward the object in the real space, deleting part of the second virtual object and/or part of the third virtual object.

17 . The method of claim 11 , wherein the second virtual object includes content information displayed in at

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Eunbin
• Park Eunyoung
• Jung Shinjae
• Lee Yoonho
• Lee Eunkyung

Classifications

• G06F3/04847

  Interaction techniques to control parameter settings, e.g. interaction with sliders or dials · CPC title

• G06F3/0482 Primary

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F1/1658

  related to the mounting of internal components, e.g. disc drive or any other functional module · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title