Identifying a specific object in a two-dimensional image of objects

Patent metadata

Field	Value
Publication number	US-2022012504-A1
Application number	US-202016927467-A
Country	US
Kind code	A1
Filing date	Jul 13, 2020
Priority date	Jul 13, 2020
Publication date	Jan 13, 2022
Grant date	—

Abstract

An object detection system can identify a specific object in a two-dimensional image. A processor can receive the two-dimensional image with bounding boxes around objects. The processor can cause a point to appear in the two-dimensional image. The point can represent a position, in three-dimensional space, of the specific object. The processor can determine an existence of a condition. The condition can be that the point is enclosed by a plurality of bounding boxes. The processor can receive, in response to the existence of the condition, a depth image. The processor can determine, in response to the existence of the condition and based on information included in the depth image and a location of the point in the two-dimensional image, a specific bounding box that encloses the point. The processor can cause, based on a location of the specific bounding box, an indication of the specific object to be presented.

First claim

1 . A system for identifying a specific object in a two-dimensional image of objects, the system comprising:
one or more processors disposed in a vehicle; a data store communicably coupled to the one or more processors and storing the two-dimensional image and a depth image; and a memory communicably coupled to the one or more processors and storing:
a communications module including instructions that when executed by the one or more processors cause the one or more processors to:
receive the two-dimensional image with bounding boxes around the objects;
receive, in response to an existence of a condition, the depth image, the condition being that a point is enclosed by a plurality of the bounding boxes, the point representing a position, in a three-dimensional space, of the specific object; and
cause, based on a location of a specific bounding box that encloses the point, an indication of the specific object to be presented;

an image processing module including instructions that when executed by the one or more processors cause the one or more processors to cause the point to appear in the two-dimensional image; and
an identification module including instructions that when executed by the one or more processors cause the one or more processors to determine, in response to the existence of the condition and based on information included in the depth image and a location of the point in the two-dimensional image, the specific bounding box that encloses the point.

2 . The system of claim 1 , wherein the identification module further includes instructions that when executed by the one or more processors cause the one or more processors to determine, in response to a lack of the existence of the condition and based on the location of the point in the two-dimensional image, the specific bounding box that encloses the point.

3 . The system of claim 1 , wherein the instructions of the communications module when executed by the one or more processors cause the one or more processors to receive, from a neural network, the two-dimensional image with the bounding boxes around the objects.

4 . The system of claim 1 , wherein the instructions of the image processing module when executed by the one or more processors cause the one or more processors to receive, from a source separate from the vehicle, information related to the point.

5 . The system of claim 4 , wherein the source separate from the vehicle comprises a cloud platform, the cloud platform storing a digital twin of the specific object.

6 . The system of claim 4 , wherein the instructions of the image processing module when executed by the one or more processors cause the one or more processors to:
receive information about a position, in the three-dimensional space, of the vehicle; receive information about a first camera that produced the two-dimensional image; and determine, based on the information related to the point, the information about the position of the vehicle, and the information about the first camera, the location of the point in the two-dimensional image.

7 . The system of claim 6 , wherein the instructions of the image processing module when executed by the one or more processors cause the one or more processors to receive the information about the position of the vehicle from another source separate from the vehicle.

8 . The system of claim 7 , wherein the other source separate from the vehicle comprises a global navigation satellite system.

9 . The system of claim 6 , wherein the information about the first camera that produced the two-dimensional image comprises at least one of an extrinsic parameter of the first camera or an intrinsic parameter of the first camera.

10 . The system of claim 6 , wherein the instructions of the communications module when executed by the one or more processors cause the one or more processors to receive the depth image from a second camera.

11 . The system of claim 10 , wherein the second camera is the first camera.

12 . The system of claim 1 , wherein the instructions of the communications module when executed by the one or more processors cause the one or more processors to at least one of:
cause the indication of the specific object to be presented on at least one of a dashboard display or a heads-up display, cause the specific object to be presented in a manner that distinguishes the specific object from another of the objects, cause the specific bounding box to be presented in a manner that distinguishes the specific bounding box from another of the bounding boxes, or cause a symbol to appear in the two-dimensional image in a vicinity of the specific object.

13 . The system of claim 1 , wherein the communications module further includes instructions that when executed by the one or more processors cause the one or more processors to:
receive, from a source separate from the vehicle, additional information associated with the specific object; and cause the additional information to be presented.

14 . The system of claim 13 , wherein the source separate from the vehicle comprises a cloud platform, the cloud platform storing a digital twin of the specific object.

15 . The system of claim 13 , wherein:
the specific object comprises another vehicle, and the additional information comprises at least one of information about the other vehicle or information about an operator of the other vehicle.

16 . The system of claim 13 , wherein the instructions of the communications module when executed by the one or more processors cause the one or more processors to cause the additional information to be presented at least one of visually or audibly.

17 . A method for identifying a specific object in a two-dimensional image of objects, the method comprising:
receiving, by a processor disposed in a vehicle, the two-dimensional image with bounding boxes around the objects; causing, by the processor, a point to appear in the two-dimensional image, the point representing a position, in a three-dimensional space, of the specific object; determining, by the processor, an existence of a condition, the condition being that the point is enclosed by a plurality of the bounding boxes; receiving, by the processor and in response to the existence of the condition, a depth image; determining, by the processor, in response to the existence of the condition, and based on information included in the depth image and a location of the point in the two-dimensional image, a specific bounding box that encloses the point; and causing, by the processor and based on a location of the specific bounding box, an indication of the specific object to be presented.

18 . The method of claim 17 , wherein the determining the specific bounding box that encloses the point comprises:
determining, from the depth image, a first distance, the first distance being between a camera that produced the depth image and a first bounding box of the plurality of the bounding boxes that enclose the point; determining, from the depth image, a second distance, the second distance being between the camera that produced the depth image and a second bounding box of the plurality of the bounding boxes that enclose the point; determining a third distance, the third distance being between the camera that produced the depth image and the position, in the three-dimensional space, of the specific object; and determining that a first difference is less than a second difference, the first difference being between the first distance and the third distance, the second differen

Assignees

• Toyota Eng & Mfg North America

Inventors

• Liu Yongkang
• Wang Ziran
• Han Kyungtae
• Shou Zhenyu
• Tiwari Prashant

Classifications

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06F18/24133

  Distances to prototypes · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title