Object detection in image data using depth segmentation

Patent metadata

Field	Value
Publication number	US-10282639-B2
Application number	US-201615363482-A
Country	US
Kind code	B2
Filing date	Nov 29, 2016
Priority date	Nov 29, 2016
Publication date	May 7, 2019
Grant date	May 7, 2019

Abstract

RGB-D data generated by at least one optical sensor for a field of view is received. Thereafter, the RGB-D data is bifurcated into (i) RGB data and (ii) depth data for the field of view. One or more bounding polygons are defined within the depth data that each characterize a window within the field of view encapsulating an object. The RGB data is then cropped using the bounding polygon(s). Image processing can later be applied to the cropped RGB data to identify at least one object therein. Related apparatus, systems, techniques and articles are also described.

First claim

What is claimed is:

1. A method for implementation by one or more data processors forming part of at least one computing device, the method comprising:
receiving, by at least one data processor, RGB-D data generated by at least one optical sensor for a field of view;
bifurcating, by at least one data processor, the RGB-D data into (i) RGB data and (ii) depth data for the field of view;

defining, by at least one data processor, at least one bounding polygon within the depth data that each characterize a window within the field of view encapsulating an object;
cropping, by at least one data processor, the RGB data using the at least one bounding polygon; and applying, by at least one data processor, image processing to the cropped RGB data to identify at least one object therein by:
applying two or more different types of bounding polygon detection techniques to the cropped RGB data to define a plurality of proposed bounding polygons; and
defining at least one final bounding polygon based on the plurality of proposed bounding polygons;
identifying and classifying the at least one object using the at least one final bounding polygon;
wherein a first bounding polygon detection technique is depth segmentation and a second bounding polygon technique is color segmentation.

2. The method of claim 1 , wherein a shape of the at least one bounding polygon is rectangular.

3. The method of claim 1 , wherein a shape of the at least one bounding polygon has three or more sides.

4. The method of claim 1 , wherein the depth segmentation groups pixels having a depth within a pre-defined range to define outer edges of the object.

5. The method of claim 1 , wherein the applying image processing comprises:
applying at least one other bounding polygon detection technique to the cropped RGB data to define at least one second bounding polygon which is used to identify and classify the object.

6. The method of claim 5 , wherein the at least one other bounding polygon detection technique comprises a technique selected from a group consisting of an edge box model, SelectiveSearch, binary normed gradients (BING), CPMC, Endres, geodesic, multi-scale combinatorial grouping (MCG), Objectness, Rahtu, randomized prism, Rantalankila, Gaussian, SlidingWindow, Superpixels, and uniform.

7. The method of claim 5 , wherein the at least one other bounding polygon detection technique comprises a color segmentation model.

8. The method of claim 1 , wherein the defining of the at least one final bounding polygon comprises:
applying an intersection over union of the plurality of proposed bounding polygons; and
selecting the at least one final bounding polygon based on which proposed bounding polygon has an area that intersects most of the other proposed bounding polygons.

9. The method of claim 8 , wherein a shape of the at least one proposed bounding polygon is rectangular.

10. The method of claim 8 , wherein a shape of the at least one proposed bounding polygon has three or more sides.

11. A method for implementation by one or more data processors forming part of at least one computing device, the method comprising:
receiving, by at least one data processor, combined color and depth data generated by at least one optical sensor for a field of view;
bifurcating, by at least one data processor, the combined color and depth data into (i) color data and (ii) depth data for the field of view;
defining, by at least one data processor, at least one bounding polygon within the depth data that each characterize a window within the field of view encapsulating an object;
cropping, by at least one data processor, the color data using the at least one bounding polygon; and
applying, by at least one data processor, image processing to the cropped color data to identify at least one object therein by:
applying two or more different types of bounding polygon detection techniques to the cropped RGB data to define a plurality of proposed bounding polygons; and
defining at least one final bounding polygon based on the plurality of proposed bounding polygons;
identifying and classifying the at least one object using the at least one final bounding polygon;
wherein a first bounding polygon detection technique is depth segmentation and a second bounding polygon technique is color segmentation.

12. The method of claim 11 , wherein the combined color and depth data is RGB-D data.

13. The method of claim 11 , wherein the combined color and depth data is point cloud data.

14. A system comprising:
at least one data processor; and
memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
receiving RGB-D data generated by at least one optical sensor for a field of view;
bifurcating the RGB-D data into (i) RGB data and (ii) depth data for the field of view;
defining, at least one bounding polygon within the depth data that each characterize a window within the field of view encapsulating an object;
cropping the RGB data using the at least one bounding polygon; and
applying image processing to the cropped RGB data to identify at least one object therein by: applying two or more different types of bounding polygon detection techniques to the cropped RGB data to define a plurality of proposed bounding polygons; and
defining at least one final bounding polygon based on the plurality of proposed bounding polygons;
identifying and classifying the at least one object using the at least one final bounding polygon;
wherein a first bounding polygon detection technique is depth segmentation and a second bounding polygon technique is color segmentation.

15. The system of claim 14 , wherein a shape of the at least one bounding polygon has three or more sides.

16. The system of claim 14 , wherein the depth segmentation groups pixels having a depth within a pre-defined range to define outer edges of the object.

17. The method of claim 1 further comprising:
generating, by at least one data processor, a segmented depth image by grouping all pixels within the depth data having depth values with a pre-defined range of values relative to one other into one of at least two groups, each group corresponding to a different depth layer including a foreground portion and a background portion; and
generating, by at least one data processor, a binary version of the segmented depth image comprising a single foreground object;
wherein the object encapsulated by the bounding polygon is the single foreground object.

18. The method of claim 11 further comprising:
generating, by at least one data processor, a segmented depth image by grouping all pixels within the depth data having depth values with a pre-defined range of values relative to one other into one of at least two groups, each group corresponding to a different depth layer including a foreground portion and a background portion; and
generating, by at least one data processor, a binary version of the segmented depth image comprising a single foreground object;
wherein the object encapsulated by the bounding polygon is the single foreground object.

19. The system of claim 14 , wherein the operations further comprise:
generating a segmented depth image by grouping all pixels within the depth data having depth values with a pre-defined range of values relative to one other into one of at least two groups, each group corresponding to a different depth layer including a foreground portion and a background portion; and
generating a binary version of the segmented depth imag

Assignees

• Sap Se

Inventors

• Farooqi Waqas Ahmad
• Lipps Jonas
• Schmidt Eckehard
• Fricke Thomas
• Verzano Nemrude

Classifications

• G06F18/24

  Classification techniques · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T7/0083

  Physics · mapped topic

• G06T2207/20132

  Image cropping · CPC title