Visual search refinement for computer generated rendering environments

Patent metadata

Field	Value
Publication number	US-11710283-B2
Application number	US-202117508010-A
Country	US
Kind code	B2
Filing date	Oct 22, 2021
Priority date	Aug 1, 2019
Publication date	Jul 25, 2023
Grant date	Jul 25, 2023

Abstract

Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

First claim

What is claimed is:

1. A method, comprising:
at a first device with a processor:
detecting a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the obtained image content or the detected feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.

2. The method of claim 1 , wherein providing the generic virtual content within the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content.

3. The method of claim 1 , wherein the generic virtual content is determined from an asset store residing on the first device.

4. The method of claim 1 , wherein the object-specific content is based on information about the physical object.

5. The method of claim 1 , wherein determining the generic virtual content associated with the physical object is based on a coarse recognition of the physical object.

6. The method of claim 1 , wherein the portion of the obtained image content excludes a second portion of the obtained image content.

7. The method of claim 1 , wherein the detected feature associated with the physical object is a shape, a color, a size, a volume, or a marker.

8. The method of claim 1 , further comprising:
determining whether the obtained image content satisfies privacy criteria; and
determining the portion of the obtained image content based on the satisfaction of the privacy criteria.

9. The method of claim 1 , further comprising:
determining whether to transmit the portion of the obtained image content based on privacy criteria.

10. The method of claim 1 , further comprising:
transmitting the detected feature to the second device.

11. The method of claim 1 , wherein the response associated with the physical object includes identification data to be used by the first device to identify the physical object in future image content.

12. The method of claim 1 , wherein the response associated with the physical object includes an assessment of a condition of the physical object.

13. The method of claim 1 , wherein the first device or the second device is a head mounted device (HMD).

14. The method of claim 1 , wherein the first device or the second device is a mobile electronic device.

15. A system comprising:
a first device with a processor; and
a computer-readable storage medium comprising instructions that upon execution by the processor cause the system to perform operations, the operations comprising:
detecting a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the obtained image content or the detected feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.

16. The system of claim 15 , wherein providing the generic virtual content within the CGR environment comprises replacing, enhancing, or supplementing the physical object with the generic virtual content.

17. The system of claim 16 , wherein the generic virtual content is determined from an asset store residing on the first device.

18. The system of claim 16 , wherein the object-specific content is based on information about the physical object.

19. The system of claim 15 , wherein determining the generic virtual content associated with the physical object is based on a coarse recognition of the physical object.

20. A non-transitory computer-readable storage medium storing program instructions that are executable by a processor to perform operations comprising:
detecting, by a first device, a physical object based on obtained image content;
determining, based on a detected feature associated with the physical object, generic virtual content associated with the physical object;
providing the generic virtual content within a computer generated rendering (CGR) environment;
transmitting a portion of the obtained image content or the detected feature to a second device;
receiving, from the second device, a response based on an identification of the physical object performed by the second device; and
updating the CGR environment by replacing the generic virtual content with object-specific content.

Assignees

• Apple Inc

Inventors

• Verma Eshan
• Ulbricht Daniel
• Blechschmidt Angela
• Baig Mohammad Haris
• Lee Chen-Yu
• Batra Tanmay

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title