Method and device for detecting objects from depth-resolved image data

Patent metadata

Field	Value
Publication number	US-9870513-B2
Application number	US-201414902176-A
Country	US
Kind code	B2
Filing date	Sep 4, 2014
Priority date	Sep 9, 2013
Publication date	Jan 16, 2018
Grant date	Jan 16, 2018

Abstract

An object-recognition method for a vehicle's driver assistance system involves obtaining a 2D image and a 3D image, forming a 3D apparent object from the 3D image, detecting one or more detected objects in a portion of the 2D image corresponding to the apparent object from the 3D image, classifying the one or more detected objects into at least one pre-defined object class, and dividing the apparent object into at least two 3D objects when the apparent object does not correspond with at least one class-specific property of the determined at least one object class.

First claim

The invention claimed is:

1. An object recognition method for a driver assistance system of a vehicle, comprising the step:
a) with a camera system of the vehicle, producing a 2D image and a 3D image of a scene outside of the vehicle;
and further comprising the following steps performed with the driver assistance system:
b) from the 3D image, forming a three-dimensionally coherent apparent object;
c) performing image evaluation of an image portion of the 2D image that corresponds to the apparent object formed from the 3D image, and thereby detecting one or more detected objects in the image portion of the 2D image;
d) performing classification of the one or more detected objects respectively into one or more pre-defined object classes;
e) assigning respective class-specific 3D dimensions respectively to the one or more detected objects based on the respective object class into which each of the one or more detected objects was classified;
f) from the 3D image, determining apparent 3D dimensions of the apparent object;
g) comparing the apparent 3D dimensions of the apparent object with the respective class-specific 3D dimensions assigned respectively to the one or more detected objects; and
h) when the comparing in the step g) indicates that the apparent object is larger than the class-specific 3D dimensions respectively assigned to the one or more detected objects, then dividing the apparent object into at least two three-dimensional objects.

2. The object recognition method according to claim 1 , further comprising the following step performed with the driver assistance system:
i) when the comparing in the step g) indicates that the apparent 3D dimensions of the apparent object correspond with the class-specific 3D dimensions assigned to a matching one of the one or more detected objects, then the apparent object is not divided and instead is verified as the matching one of the one or more detected objects.

3. The object recognition method according to claim 2 , further comprising, with the driver assistance system performing a driver assistance function based on the matching one of the one or more detected objects.

4. The object recognition method according to claim 1 , further comprising, with the driver assistance system performing a driver assistance function based on the at least two three-dimensional objects.

5. The object recognition method according to claim 1 , wherein the dividing of the apparent object into the at least two three-dimensional objects is performed based on the class-specific 3D dimensions assigned to the one or more detected objects.

6. The object recognition method according to claim 1 , wherein:
the camera system comprises a 3D camera and a 2D camera,
the 3D camera comprises a stereo camera, a time-of-flight camera, or a photonic mixer device,
the 3D camera produces the 3D image,
the 2D camera produces the 2D image,
the scene outside the vehicle represents surroundings outside the vehicle, and
the 3D image and the 2D image represent at least partially overlapping areas of the surroundings.

7. The object recognition method according to claim 6 , wherein the 3D camera comprises the stereo camera which includes two monocular camera sensors, and wherein the 2D camera comprises one of the two monocular camera sensors of the stereo camera.

8. The object recognition method according to claim 1 , further comprising determining at least one 3D placeholder according to a result of the classification of the one or more detected objects, and taking the at least one 3D placeholder into account as at least one replacement for the apparent object in the 3D image.

9. The object recognition method according to claim 8 , wherein the at least one 3D placeholder respectively is a frustum.

10. The object recognition method according to claim 8 , wherein the at least one 3D placeholder respectively takes into account tolerances resulting from the object forming in the step b) and/or the image evaluation and object detection in the step c).

11. The object recognition method according to claim 8 , wherein the at least one 3D placeholder respectively takes into account a spread range of 3D dimensions within the class-specific 3D dimensions of each respective one of the one or more pre-defined object classes.

12. The object recognition method according to claim 8 , further comprising comparing the at least one 3D placeholder with the apparent object, and when a matching one of the at least one 3D placeholder corresponds approximately with the apparent object then the apparent object is not divided and instead is verified as corresponding to one of the detected objects represented by the matching one of the at least one 3D placeholder.

13. The object recognition method according to claim 8 , further comprising forming a three-dimensionally coherent further apparent object from the 3D image while taking into account the at least one 3D placeholder, wherein the forming of the further apparent object beyond spatial limits of the at least one 3D placeholder is made difficult.

14. A device for performing the method according to claim 1 , comprising:
a 3D camera of the camera system configured and arranged to produce the 3D image,
a camera sensor of the camera system configured and arranged to produce the 2D image,
a first object forming unit configured and arranged to perform the step b), and
an image evaluation and classification device configured and arranged to perform the steps c), d), e), f), g) and h).

15. An object recognition method for a driver assistance system of a vehicle, comprising the following steps performed with the driver assistance system:
forming at least one 3D object from a depth image of a 3D camera,
evaluating and classifying at least one 2D object in a 2D image which corresponds to the at least one 3D object formed from the depth image,
evaluating and classifying the at least one 3D object, which comprises dividing the at least one 3D object into a plurality of individual 3D objects, when at least one property of the at least one 3D object does not match a corresponding property determined by the classification of the at least one 2D object in the 2D image, and
determining at least one 3D placeholder according to a result of the classifying of the at least one 2D object, and taking the at least one 3D placeholder into account as respectively at least one replacement for the at least one 3D object in the depth image.

16. The object detection method according to claim 15 , wherein the 3D placeholder is a frustum.

17. The object detection method according to claim 15 , wherein the 3D placeholder takes into account tolerances resulting from the 3D and/or 2D image detection and evaluation.

18. The object detection method according to claim 15 , wherein the 3D placeholder takes into account a spread of 3D dimensions within a class of objects.

19. The object detection method according to claim 15 , further comprising comparing the 3D placeholder with the at least one 3D object and, if the 3D placeholder corresponds approximately to the 3D object, then the 3D object is not divided.

20. The object detection method according to claim 15 , further comprising forming at least one renewed 3D object from the depth image while taking into account the at least one 3D placeholder, wherein object forming beyond limits of the 3D placeholder is made difficult.

Assignees

• Conti Temic Microelectronic Gmbh

Inventors

• Thiel Robert
• Bachmann Alexander

Classifications

• G06F18/24

  Classification techniques · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/204

  using stereoscopic image cameras (stereoscopic photography G03B35/00) · CPC title

• H04N13/275

  from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals · CPC title

• B60R2300/8093

  for obstacle warning · CPC title