System and method for object modification using mixed reality

Patent metadata

Field	Value
Publication number	US-10646022-B2
Application number	US-201715850310-A
Country	US
Kind code	B2
Filing date	Dec 21, 2017
Priority date	Dec 21, 2017
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

A method includes retrieving, by a device, contextual information based on at least one of an image, the device, user context, or a combination thereof. At least one model is identified from multiple models based on the contextual information and at least one object recognized in an image based on at least one model. At least one icon is displayed at the device. The at least one icon being associated with at least one of an application, a service, or a combination thereof providing additional information.

First claim

What is claimed is:

1. A smart mirror device comprising:
a memory that stores instructions; and
a processor that executes the instructions to:
receive first information associated with a superimposed heat map that is mapped to a three-dimensional mask for an object, wherein the heat map uses variable information to provide visual indications of different features of the object, the visual indications indicating at least one trimming setting in each region where an electronic device is directed to operate on the object, and the at least one trimming setting in each region is sent to the electronic device through wireless communication and controls the electronic device by adjustment of a height setting on the electronic device made based on the wireless communication;
receive second information for detection of contact of the electronic device with the object; and
provide a first communication to the electronic device based on determining position of the electronic device in relation to the object and the superimposed heat map.

2. The smart mirror device of claim 1 , further comprising:
a mirror overlayed on a display panel;

wherein:
the first communication is provided to a position and orientation tracking unit for tracking six degrees of freedom (6DOF) positions and orientations of the electronic device;
the first communication causes at least one of haptic indications, visual indications, sound or a combination thereof; and
the height setting is adjusted by modifying height of at least one of a blade, a comb or a brush on the electronic device.

3. The smart mirror device of claim 2 , wherein:
the processor further executes the instructions to receive communications, in real-time, from the electronic device for object information comprising: tracked 6DOF positions and orientations; and
the smart mirror device operates in at least one state with mirroring functionality for reflecting the visual indications.

4. The smart mirror device of claim 3 , wherein:
the processor further executes the instructions to: receive a template selection, control the electronic device based on a selected template, and superimpose the heat map upon a reflected image of the object for identifying a difference between the selected template and the reflected image of the object.

5. The smart mirror device of claim 4 , wherein:
the first communication causes the electronic device to estimate tracking deviations and use kinematics to determine counter movements for an actuator and motor of the electronic device to reduce the deviations; and
the processor further executes the instructions to use features of the object to build the three-dimensional mask of the object to map the selected template to the object and features of the object.

6. The smart mirror device of claim 5 , wherein:
the smart mirror device is coupled to an image capturing device for tracking the features of the object; and
the processor further executes the instructions to provide an operation path for the electronic device to reach a desired effect using the selected template, and display the difference by the smart mirror device based on the tracked features.

7. The smart mirror device of claim 4 , wherein:
the first communication stops when the electronic device rotates to a desired orientation and position based on the selected template; and
the smart mirror device causes a robotic element of the electronic device to adjust position and orientation based on relative position of the object and the heat map.

8. The smart mirror device of claim 7 , wherein the robotic element comprises one of: a single color or multiple color ink deploying device, or a trimmer device coupled to a comb device that is controlled by the actuator.

9. The smart mirror device of claim 4 , wherein:
template information is shareable across social media; and
the selected template is modifiable for storing as a new template.

10. The smart mirror device of claim 1 , wherein:
the variable information comprises at least one of: color information or shading information;
the visual indications comprise color and shading;
the features of the object comprise facial information; and
consolidated information from other devices connected with the smart mirror device provide collected health related information for cross correlation to observe patterns and interdependencies for planning template selection and receiving communications for template selection.

11. A method comprising:
receiving first information, by a first electronic device, for a superimposed heat map that is mapped to a three-dimensional mask for an object, wherein the heat map uses variable information to provide visual indications of different features of the object, the visual indications indicating at least one trimming setting in each region where a second electronic device is directed to operate on the object, and the at least one trimming setting in each region is sent to the second electronic device through wireless communication and controls the second electronic device by adjustment of a height setting on the second electronic device made based on the wireless communication;
receiving second information for detection of contact of the second electronic device with the object; and
providing a first communication to the second electronic device based on determining position of the second electronic device in relation to the object and the superimposed heat map.

12. The method of claim 11 , further comprising:
providing the first communication to a position and orientation tracking unit for tracking six degrees of freedom (6DOF) positions and orientations of the second electronic device; and
causing, by the first communication, at least one of haptic indications, visual indications, sound or a combination thereof;
wherein the variable information comprises at least one of: color information or shading information, the visual indications comprise color and shading, and the height setting is adjusted by modifying height of at least one of a blade, a comb or a brush on the second electronic device.

13. The method of claim 12 , further comprising:
receiving communications, in real-time, from the second electronic device for object information comprising: tracked 6DOF positions and orientations; and
operating, by the first electronic device, in at least one state with mirroring functionality for reflecting the visual indications.

14. The method of claim 13 , further comprising:
receiving, by the first electronic device, a template selection, to control the second electronic device based on a selected template; and
superimposing the heat map upon a reflected image of the object to identify a difference between the selected template and the reflected image of the object.

15. The method of claim 14 , further comprising:
sending the first communication to the second electronic device to cause the second electronic device to estimate tracking deviations and use kinematics to determine counter movements for an actuator and motor of the second electronic device to reduce the deviations; and
building, based on features of the object, the three-dimensional mask of the object to map the selected template to the object and features of the object.

16. The method of claim 15 , further comprising:
tracking the features of the object using an image capturing device;
providing an operation path for the second electronic device to reach a desired effect using the selected template; and
displaying the difference by the first device based on the tracked features.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Yoganandan Arun Rakesh
• Mcgill Kyle T
• ZHU JIN Chang Long
• Hibino Stacie L
• Akiyoshi Kumi

Classifications

• A45D44/005 Primary

  for selecting or displaying personal cosmetic colours or hairstyle · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B26B21/4056

  Sensors or controlling means · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title