Fast recognition algorithm processing, systems and methods

Patent metadata

Field	Value
Publication number	US-10628673-B2
Application number	US-201815881043-A
Country	US
Kind code	B2
Filing date	Jan 26, 2018
Priority date	Jul 19, 2013
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

Systems and methods of quickly recognizing or differentiating many objects are presented. Contemplated systems include an object model database storing recognition models associated with known modeled objects. The object identifiers can be indexed in the object model database based on recognition features derived from key frames of the modeled object. Such objects are recognized by a recognition engine at a later time. The recognition engine can construct a recognition strategy based on a current context where the recognition strategy includes rules for executing one or more recognition algorithms on a digital representation of a scene. The recognition engine can recognize an object from the object model database, and then attempt to identify key frame bundles that are contextually relevant, which can then be used to track the object or to query a content database for content information.

First claim

What is claimed is:

1. An autonomous vehicle apparatus comprising:
at least one sensor;
at least one hardware processor;
a tangible, non-transitory computer readable memory storing an object model database and software instructions; and
a recognition engine, executable on the at least one hardware processor, coupled with the memory and the at least one sensor, and configurable, upon execution of the software instructions, to:
capture, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;
obtain access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;
track recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at least one of another scene object and background of the scene;
recognize a scene object as the at least one known object using at least one recognition algorithm, the recognition features, and the digital representation; and
initiate a vehicle action responsive to the scene object.

2. The apparatus of claim 1 , wherein the vehicle action comprises interpreting a gesture of a scene object as at least one of a command, a control, or a communication.

3. The apparatus of claim 1 , wherein the vehicle action comprises obtaining access to content associated with the at least one known object.

4. The apparatus of claim 3 , wherein the content comprises at least one of a link, a URL, image data, video data, audio data, augmented reality data, social media data, product data, text data, object data, object model data, game data, news data, multimedia data, and content information corresponding to the at least one known object.

5. The apparatus of claim 1 , wherein the recognition engine is further configurable to recognize a plurality of scene objects in parallel.

6. The apparatus of claim 1 , wherein the recognition engine is further configurable to analyze or interpret one or more scene objects.

7. The apparatus of claim 1 , wherein obtaining access to the contextually relevant key frame bundles is triggered by one or more of a sound, location information, time, user profile information, or other non-visual information.

8. The apparatus of claim 1 , wherein the contextually relevant key frame bundles are pre-cached in the memory based on vehicle position or location data obtained from the at least one sensor device.

9. The apparatus of claim 1 , wherein key frame bundles comprise information related to one or more magnetic, optical, thermal, acoustic, chemical, and electrical properties of a modeled object.

10. The apparatus of claim 1 , wherein the key frame bundles are contextually relevant to a current context of the recognition engine based on one or more of a location or motion of a vehicle.

11. The apparatus of claim 1 , wherein the digital representation comprises at least one of image data, video data, audio data, location data, biometric data, tactile data, time data, temperature data, weather data, and accelerometer data.

12. The apparatus of claim 1 , wherein the digital representation comprises multi-dimensional data.

13. The apparatus of claim 1 , wherein the context comprises at least one of time information, location information, orientation information, ambient audio information, user profile information, directional information, weather information, proximity information, temperature information, tracking information, and acceleration information.

14. The apparatus of claim 1 , wherein the context includes at least one of a contextual device state, a shopping context, a commute context, and a non-visual context.

15. The apparatus of claim 1 , wherein the at least one known object comprises at least one of a building, an automobile or vehicle, a person, a face, a moveable object, and a street fixture.

16. The apparatus of claim 1 , wherein the recognition engine comprises a cloud-based recognition service.

17. The apparatus of claim 1 , wherein the at least one sensor comprises at least one of a GPS device, hall probe, camera, RFID reader, near field radio, microphone, biometric sensor, touch screen, accelerometer, magnetometer, gyroscope, spectrometer, strain or stress gauge, pulse oximeter, seismometer, galvanometer, Radar, LIDAR, infrared sensor, flow sensor, anemometer, Geiger counter, scintillator, barometer, and piezoelectric sensor.

18. The apparatus of claim 1 , wherein the at least one recognition algorithm comprises an algorithm prioritizing execution of a plurality of processing algorithms based on the context derived from the digital representation.

19. The apparatus of claim 18 , wherein prioritizing execution is based on one or more of a time, a location, an orientation, a contextual parameter, a GPS coordinate or position, a user, a license agreement, a digital representation attribute, a frame rate, and a hierarchy.

20. The apparatus of claim 18 , wherein one of a plurality of processing algorithms is selected to process data from the digital representation based on a processing result corresponding to another one of the plurality of processing algorithms.

21. The apparatus of claim 1 , wherein object models in the object model database are constructed from scanning real-world objects using one or more of a laser beam, LIDAR, a time-of-flight system, and a structured light system.

22. An autonomous vehicle method comprising:
capturing, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;
obtaining access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;
tracking recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at least one of another scene object and background of the scene;
recognizing a scene object as the at least one known object using at least one recognition algorithm, the recognition features, and the digital representation; and
initiating a vehicle action responsive to the scene object.

23. A non-transitory computer readable medium or media containing instructions for executing a method comprising:
capturing, via the at least one sensor, a digital representation a scene comprising a plurality of scene objects in a real-world environment;
obtaining access to contextually relevant key frame bundles based on a context derived from the digital representation, wherein the contextually relevant key frame bundles correspond to recognition features associated with modeled features of at least one known object;
tracking recognized scene objects relative to each other in real-time based on information in the contextually relevant key frame bundles and as a function of the recognition features, wherein the tracking includes differentiating at least one scene object from at

Assignees

• Nant Holdings Ip Llc

Inventors

• Wnuk Kamil
• Song Bing
• Siddiqui Matheen
• Mckinnon David
• Sudol Jeremi
• Soon-Shiong Patrick
• Dialameh Orang

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20004

  Adaptive image processing · CPC title

• G16H30/40

  for processing medical images, e.g. editing · CPC title

• G06T7/0014

  using an image reference approach · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title