Visual search refinement for computer generated rendering environments

Patent metadata

Field	Value
Publication number	US-11189103-B2
Application number	US-202016924559-A
Country	US
Kind code	B2
Filing date	Jul 9, 2020
Priority date	Aug 1, 2019
Publication date	Nov 30, 2021
Grant date	Nov 30, 2021

Abstract

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

First claim

What is claimed is:

1. A method, comprising:
at a first device with a processor, a display, and a camera:
obtaining an image or video of a physical setting via the camera;
classifying a physical object into a category based on a feature associated with the physical object that is detected in the image or video;
providing a computer generated rendering (CGR) environment on the display, the CGR environment comprising first virtual content associated with the physical object, the first virtual content comprising generic virtual content that is based on the category;
transmitting a portion of the image or video, the feature, or the category to a second device;
receiving a response associated with the physical object, the response determined based on an identification of the physical object performed by the second device; and
updating the CGR environment on the display, wherein updating the CGR environment comprises replacing the first virtual content with second virtual content based on the response, wherein the second virtual content comprises object-specific content based on the response.

2. The method of claim 1 , wherein providing the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content, wherein the generic virtual content is from an asset store residing on the first device based on the category.

3. The method of claim 2 , wherein the object-specific content is based on information about the physical object.

4. The method of claim 1 , wherein classifying the physical object into the category is based on a coarse recognition of the physical object.

5. The method of claim 1 , wherein the portion of the image or video excludes a second portion of the image or video.

6. The method of claim 1 , wherein the feature is a shape, a color, a size, a volume, or a marker.

7. The method of claim 1 , wherein the response includes an identification of a sub-category, an identification of a particular type of item, or supplemental information.

8. The method of claim 1 , further comprising:
determining whether the image or video satisfies privacy criteria; and
determining the portion of the image or video based on the satisfaction of the privacy criteria.

9. The method of claim 1 , further comprising determining whether to transmit the portion of the image or video based on privacy criteria.

10. The method of claim 1 , further comprising transmitting the category to the second device.

11. The method of claim 1 , wherein the received response associated with the physical object includes identification data to be used by the first device to identify the physical object in a future image or video.

12. The method of claim 1 , wherein the response associated with the physical object includes an assessment of a condition of the physical object.

13. The method of claim 1 , wherein the first device is a head mounted device (HMD).

14. The method of claim 1 , wherein the first device is a mobile electronic device.

15. A system comprising:
a first device with a display and a camera;
a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
obtaining an image or video of a physical setting via the camera
classifying a physical object into a category based on a feature associated with the physical object that is detected in the image or video;
providing a computer generated rendering (CGR) environment on the display, the CGR environment comprising first virtual content associated with the physical object, the first virtual content comprising generic virtual content that is based on the category;
transmitting a portion of the image or video, the feature, or the category to a second device;
receiving a response associated with the physical object, the response determined based on an identification of the physical object performed by the second device; and
updating the CGR environment on the display, wherein updating the CGR environment comprises replacing the first virtual content with second virtual content based on the response, wherein the second virtual content comprises object-specific content based on the response.

16. The system of claim 15 , wherein providing the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content, wherein the generic virtual content is from an asset store residing on the first device based on the category.

17. The system of claim 16 , wherein the object-specific content is based on information about the physical object.

18. The system of claim 15 , wherein classifying the physical object into the category is based on a coarse recognition of the physical object.

19. The system of claim 15 , wherein the portion of the image or video excludes a second portion of the image or video.

20. A non-transitory computer-readable storage medium storing program instructions that are executable by a processor to perform operations comprising:
obtaining, by a first device including a camera and a display, an image or video of a physical setting via the camera;
classifying, by the first device, a physical object into a category based on a feature associated with the physical object that is detected in the image or video;
providing a computer generated rendering (CGR) environment on the display, the CGR environment comprising first virtual content associated with the physical object, the first virtual content comprising generic virtual content that is based on the category;
transmitting a portion of the image or video, the feature, or the category to a second device;
receiving a response associated with the physical object, the response determined based on an identification of the physical object performed by the second device; and
updating the CGR environment on the display, wherein updating the CGR environment comprises replacing the first virtual content with second virtual content based on the response, wherein the second virtual content comprises object-specific content based on the response.

Assignees

• Apple Inc

Inventors

• Verma Eshan
• Ulbricht Daniel
• Blechschmidt Angela
• Baig Mohammad Haris
• Lee Chen-Yu
• Batra Tanmay

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/14

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title