Object ingestion and recognition systems and methods

Patent metadata

Field	Value
Publication number	US-11748990-B2
Application number	US-202217830252-A
Country	US
Kind code	B2
Filing date	Jun 1, 2022
Priority date	Feb 14, 2014
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

An object recognition ingestion system is presented. The object ingestion system captures image data of objects, possibly in an uncontrolled setting. The image data is analyzed to determine if one or more a priori know canonical shape objects match the object represented in the image data. The canonical shape object also includes one or more reference PoVs indicating perspectives from which to analyze objects having the corresponding shape. An object ingestion engine combines the canonical shape object along with the image data to create a model of the object. The engine generates a desirable set of model PoVs from the reference PoVs, and then generates recognition descriptors from each of the model PoVs. The descriptors, image data, model PoVs, or other contextually relevant information are combined into key frame bundles having sufficient information to allow other computing devices to recognize the object at a later time.

First claim

What is claimed is:

1. An object recognition and ingestion system, comprising:
at least one non-transitory computer readable memory storing executable object recognition and ingestion software instructions; and
at least one processor coupled with the at least one non-transitory computer readable memory that, upon execution of the object recognition and ingestion software instructions, performs operations to:
obtain a digital representation of a scene, wherein the digital representation is obtained from at least one sensor and further includes image data of at least one three-dimensional object and location information;
obtain a result set of shape objects from a set of one or more candidate shape objects, wherein the result set includes at least one shape object from the set of one or more candidate shape objects and that has at least one shape attribute satisfying selection criteria determined from geometrical information of the at least one three-dimensional object derived from the image data of the at least one three-dimensional object;
select at least one target shape object from the result set of shape objects based on a context and at least one point-of-view associated with the at least one three-dimensional object;
instantiate at least one three-dimensional object model of the at least one three-dimensional object from the at least one target shape object and the image data; and
store, in an object recognition database, a bundle of recognition parameters derived from the object model and location information, wherein the recognition parameters enable a computing device to recognize the at least one three-dimensional object.

2. The system of claim 1 , wherein the at least one shape object comprises a simple shape.

3. The system of claim 1 , wherein the at least one shape object comprises a compound shape.

4. The system of claim 1 , wherein the at least one shape object comprises a geometric primitive.

5. The system of claim 1 , wherein the at least one shape object comprises an object template.

6. The system of claim 5 , wherein the object template includes at least one of the following: a tree, a landmark, and a building.

7. The system of claim 5 , wherein the object template includes at least one of the following: a car, a plane, a human, an appliance, and a toy.

8. The system of claim 1 , wherein the location information comprises a GPS location.

9. The system of claim 1 , wherein the location information comprises a location of the at least one three-dimensional object.

10. The system of claim 1 , wherein the operations further include updating the object recognition database with new information related to the at least one three-dimensional object over time.

11. The system of claim 10 , wherein the new information includes new image data integrated into the bundle of recognition parameters of the at least one three-dimensional object.

12. The system of claim 1 , wherein the at least one shape object comprises a triangle.

13. The system of claim 1 , wherein the at least one object comprises at least one of the following: a landmark and a tourist attraction.

14. The system of claim 1 , wherein the at least one shape object in the result set of shape objects also satisfies the selection criteria based on the location information.

15. The system of claim 1 , wherein the context includes at least one of: a location, a time, a temperature, a weather condition, a recognized object, and an orientation.

16. The system of claim 1 , wherein the context comprises a positive association with respect to the location information.

17. The system of claim 1 , wherein the context comprises a negative association with respect to the location information.

18. The system of claim 1 , wherein the selection criteria depend on edge descriptors derived from the image data.

19. The system of claim 1 , wherein the computing device comprises at least one of: a video recording device, a head mounted visor; head-mounted glasses, a game console, a game interface, a webcam, and a smart phone.

20. The system of claim 1 , wherein the recognition parameters include metadata comprising contextual recognition information.

21. The system of claim 20 , wherein the contextual recognition information includes at least one of: a location, a time, a user identity, and a weather condition.

22. The system of claim 1 , wherein the one or more candidate shape objects are indexed.

23. An object ingestion and recognition method comprising:
obtaining a digital representation of a scene, wherein the digital representation is obtained from at least one sensor and further includes image data of at least one three-dimensional object and location information;
obtaining a result set of shape objects from a plurality of candidate shape objects, wherein the result set includes at least one shape object from the plurality of candidate shape objects and that has at least one shape attribute satisfying selection criteria determined from geometrical information of the at least one three-dimensional object derived from the image data of the at least one three-dimensional object;
selecting at least one target shape object from the result set of shape objects based on a context and at least one point-of-view associated with the at least one three-dimensional object;
instantiating at least one three-dimensional object model of the at least one three-dimensional object from the at least one target shape object and the image data; and
storing, in an object recognition database, a bundle of recognition parameters derived from the three-dimensional object model and location information, wherein the recognition parameters enable a computing device to recognize the at least one three-dimensional object.

24. A non-transitory computer readable medium storing one or more executable instructions for ingesting and recognizing one or more objects, which when executed by at least one processor coupled to the non-transitory computer readable medium perform:
obtaining a digital representation of a scene, wherein the digital representation is obtained from at least one sensor and further includes image data of at least one three-dimensional object and location information;
obtaining a result set of shape objects from a plurality of candidate shape objects, wherein the result set includes at least one shape object from the plurality of candidate shape objects and that has at least one shape attribute satisfying selection criteria determined from geometrical information of the at least one three-dimensional object derived from the image data of the at least one three-dimensional object;
selecting at least one target shape object from the result set of shape objects based on a context and at least one point-of-view associated with the at least one three-dimensional object;
instantiating at least one three-dimensional object model of the at least one three-dimensional object from the at least one target shape object and the image data; and
storing, in an object recognition database, a bundle of recognition parameters derived from the three-dimensional object model and location information, wherein the recognition parameters enable a computing device to recognize the at least one three-dimensional object.

Assignees

• Nant Holdings Ip Llc

Inventors

• Wnuk Kamil
• Mckinnon David
• Sudol Jeremi
• Song Bing
• Siddiqui Matheen

Classifications

• G06V20/46 Primary

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06F16/532

  Query formulation, e.g. graphical querying · CPC title

• G06F16/5838

  using colour · CPC title

• G06F16/5854

  using shape and object relationship · CPC title

• G06F16/5866

  using information manually generated, e.g. tags, keywords, comments, manually generated location and time information · CPC title