System and method for detecting errors and improving reliability of perception systems using logical scaffolds

Patent metadata

Field	Value
Publication number	US-11157756-B2
Application number	US-202016745560-A
Country	US
Kind code	B2
Filing date	Jan 17, 2020
Priority date	Aug 19, 2019
Publication date	Oct 26, 2021
Grant date	Oct 26, 2021

Abstract

An artificial intelligence perception system for detecting one or more objects includes one or more processors, at least one sensor, and a memory device. The memory device includes an image capture module, an object identifying module, and a logical scaffold module. The image capture module and the object identifying module cause the one or more processors to obtain sensor information of a field of view from a sensor, identify an object within the sensor information, and determine at least one property of the object. The logical scaffold module causes the one or more processors to determine, by a logical scaffold, when the at least one property of the object as determined by the object identifying module is one of a true condition or a false condition.

First claim

What is claimed is:

1. An artificial intelligence perception system for detecting one or more objects comprising:
one or more processors;
at least one sensor operably connected to the one or more processors, the at least one sensor having a field of view of an environment; and
a memory device operably connected to the one or more processors, the memory device comprising:
an image capture module having instructions that when executed by the one or more processors causes the one or more processors to obtain sensor information of the field of view from the at least one sensor,
an object identifying module having instructions that when executed by the one or more processors causes the one or more processors to identify, using a trained perception model, an object within the sensor information and determine at least one property of the object based on the sensor information, and
a logical scaffold module having instructions that when executed by the one or more processors causes the one or more processors to determine when the at least one property of the object as determined by trained perception model is one of a true condition or a false condition, wherein the true condition indicates that the at least one property of the object as determined by the trained perception model satisfies a logical criterion, indicating that the at least one property of the object is correct based on a discrete requirement stored in a data store, and the false condition indicates that the at least one property of the object as determined by the trained perception model fails the logical criterion, indicating that the at least one property of the object is incorrect based on the discrete requirement.

2. The artificial intelligence perception system of claim 1 , wherein the at least one property of the object as determined by the trained perception model includes at least one of a temporal property, an ontological property, and a relationship related property.

3. The artificial intelligence perception system of claim 2 , wherein the ontological property of the object includes at least one of a motion of the object, a velocity of the object, a location of the object, a transition of an object from one type of object to another type of object.

4. The artificial intelligence perception system of claim 2 , wherein the temporal property of the object includes at least one of a time the object is within the sensor information and a consistency of the object within the sensor information.

5. The artificial intelligence perception system of claim 2 , wherein the relationship related property of the object includes a spatial relationship between the object and another object.

6. The artificial intelligence perception system of claim 1 , wherein the memory device further comprises an output module having instructions that when executed by the one or more processors causes the one or more processors to output an indicator when the at least one property of the object as determined by the trained perception model fails the logical criterion, wherein the indicator includes the sensor information used to determine the at least one property of the object.

7. A method for detecting one or more objects by an artificial intelligence perception system, the method comprising the steps of:
obtaining sensor information of a field of view from a sensor;
identifying, by the artificial intelligence perception system, an object within the sensor information;
determining, by the artificial intelligence perception system using a trained perception model, at least one property of the object based on the sensor information;
determining, by a logical scaffold, when the at least one property of the object as determined by the trained perception model is one of a true condition or a false condition; and
wherein the true condition indicates that the at least one property of the object as determined by the artificial intelligence perception system satisfies a logical criterion, indicating that the at least one property of the object is correct based on a discrete requirement stored in a data store, and the false condition indicates that the at least one property of the object as determined by the trained perception model fails the logical criterion indicating that the at least one property of the object is incorrect based on the discrete requirement.

8. The method of claim 7 , wherein the at least one property of the object as determined by the trained perception model includes at least one of a temporal property, an ontological property, and a relationship related property.

9. The method of claim 8 , wherein the ontological property of the object includes at least one of a motion of the object, a velocity of the object, a location of the object, a transition of an object from one type of object to another type of object.

10. The method of claim 8 , wherein the temporal property of the object includes at least one of a time the object is within the sensor information and a consistency of the object within the sensor information.

11. The method of claim 8 , wherein the relationship related property of the object includes a spatial relationship between the object and another object.

12. The method of claim 7 , further comprising the steps of outputting an indicator when the at least one property of the object as determined by the trained perception model fails the logical criterion, wherein the indicator includes the sensor information used to determine the at least one property of the object.

13. The method of claim 12 , further comprising the step of retraining the trained perception model using the sensor information that resulted in the false condition.

14. The method of claim 7 , wherein the logical scaffold is written in temporal logic.

15. A non-transitory computer-readable medium for detecting one or more objects by an artificial intelligence perception system, the non-transitory computer-readable medium comprising instructions that when executed by one or more processors cause the one or more processors to:
obtain sensor information of a field of view from a sensor;
identify, by the artificial intelligence perception system using a trained perception model, an object within the sensor information;
determine, by the trained perception model, at least one property of the object based on the sensor information;
determine, by a logical scaffold, when the at least one property of the object as determined by the trained perception model is one of a true condition or a false condition; and
wherein the true condition indicates that the at least one property of the object as determined by the trained perception model satisfies a logical criterion, indicating that the at least one property of the object is correct based on a discrete requirement stored in a data store, and the false condition indicates that the at least one property of the object as determined by the trained perception model fails the logical criterion indicating that the at least one property of the object is incorrect based on the discrete requirement.

16. The non-transitory computer-readable medium of claim 15 , wherein the at least one property of the object as determined by the trained perception model includes at least one of a temporal property, an ontological property, and a relationship related property.

17. The non-transitory computer-readable medium of claim 16 , wherein the ontological property of the object includes at least one of a motion of the object, a velocity of the object, a location of the object, a transition of an object from one type of object

Assignees

• Toyota Res Inst Inc

Inventors

• Arechiga Gonzalez Nikos
• Kong Soonho
• Decastro Jonathan
• Behere Sagar
• Park Dennis

Classifications

• G06V20/582

  of traffic signs · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title