Method and apparatus for image segmentation using an event sensor

Patent metadata

Field	Value
Publication number	US-11263751-B2
Application number	US-201916666225-A
Country	US
Kind code	B2
Filing date	Oct 28, 2019
Priority date	Oct 26, 2018
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

A method for real-time semantic image segmentation using a monocular event-based sensor includes capturing a scene using a red, green, blue (RGB) sensor to obtain a plurality of RGB frames and an event sensor to obtain event data corresponding to each of the plurality of RGB frames, performing object labeling for objects in a first RGB frame among the plurality of RGB frames by identifying one or more object classes, obtaining an event velocity of the scene by fusing the event data corresponding to the first RGB frame and at least one subsequent RGB frame among the plurality of RGB frames, determining whether the event velocity is greater than a predefined threshold, and performing object labeling for objects in the at least one subsequent RGB frame based on the determination.

First claim

What is claimed is:

1. A method for image segmentation in an electronic device, the method comprising:
obtaining a first RGB frame of a scene using a sensor, the sensor comprising:
a red, green, blue (RGB) sensor generating RGB data, and
an event sensor generating event data corresponding to the first RGB frame;

classifying objects in the scene to one or more classes of objects by identifying the one or more classes of objects in the scene;
obtaining event data corresponding to a second RGB frame with the event sensor at multiple sensitivity levels by changing exposure values of the event sensor;
identifying motion information of the scene with the classified objects based on the first RGB frame and the event data of the second RGB frame;
predicting an object boundary of the second RGB frame using the one or more classes of objects in the scene; and
estimating the object boundary for the second RGB frame using the motion information.

2. The method of claim 1 , wherein the event data of the scene is associated with motions of objects in the scene.

3. The method of claim 1 , wherein the event data is updated with events that are detected by changes of pixel intensity in an image of the scene.

4. The method of claim 3 , further comprising:
accumulating the events over a period of time; and
correlating the accumulated events with RGB frames including the first RGB frame and the second RGB frame.

5. The method of claim 4 , further comprising:
determining an optimum sensitivity level of the event sensor among multiple sensitivity levels to identify spatially inconsistent regions based on the event data including the accumulated events.

6. The method of claim 4 , wherein the identifying of the motion information comprises identifying the motion information of the scene with the accumulated events.

7. The method of claim 4 , wherein the classifying of the objects in the scene to the one or more classes of objects comprises:
generating an event velocity of the scene with changes in the accumulated events;
determining whether the event velocity is greater than a predefined threshold value; and
classifying the objects in the scene to the one or more classes of objects in response to the determination.

8. The method of claim 1 , further comprising:
identifying spatially inconsistent regions in the scene in response to obtaining the event data by changing sensitivity levels of the event sensor; and
masking the identified spatially inconsistent regions in an image of the scene.

9. The method of claim 1 , wherein the identifying of the motion information of the scene comprises identifying the motion information of the scene by detecting changes in the object boundary with the event data of the second RGB frame.

10. The method of claim 1 , wherein the event sensor captures the scene at a higher frame rate than the RGB sensor.

11. A method for image segmentation in an electronic device, the method comprising:
capturing a scene using a red, green, blue (RGB) sensor to obtain a plurality of RGB frames and an event sensor to obtain event data corresponding to each of the plurality of RGB frames;
performing object labeling for objects in a first RGB frame among the plurality of RGB frames by identifying one or more object classes;
obtaining an event velocity of the scene by fusing the event data corresponding to the first RGB frame and at least one subsequent RGB frame among the plurality of RGB frames;
determining whether the event velocity is greater than a predefined threshold; and
performing object labeling for objects in the at least one subsequent RGB frame based on the determination,
wherein the performing of the object labeling comprises classifying the objects into the one or more object classes.

12. The method of claim 11 , further comprising:
identifying at least one object class in the captured scene during the performing of the object labeling for the objects in the first RGB frame;
predicting at least one object class in the at least one subsequent RGB frame using the at least one object labeling in the first RGB frame; and
updating the at least one object class of the at least one subsequent RGB frame using the event velocity of the scene.

13. The method of claim 12 , further comprising:
extracting at least one feature from the scene using the event data accumulated with the first RGB frame and the at least one subsequent RGB frame;
correlating the at least one feature with the at least one subsequent RGB frame to identify spatially inconsistent regions in the scene; and
correcting the prediction of the at least one object class using the spatially inconsistent regions.

14. The method of claim 13 , wherein the extracting of the at least one feature comprises extracting the at least one feature using the event data obtained at different sensitivity levels of the event sensor.

15. The method of claim 11 , further comprising:
generating the event data at a plurality of sensitivity levels by changing exposure values of the event sensor.

16. An apparatus for image segmentation, the apparatus comprising:
a sensor unit configured to obtain a first red, green, blue (RGB) frame of a scene, the sensor unit comprising:
an RGB sensor configured to generate RGB data, and
an event sensor configured to generate event data of the first RGB frame and a second RGB frame, wherein the event sensor is configured to obtain the second RGB frame at multiple sensitivity levels by changing exposure values of the event sensor; and

a processor configured to:
classify objects in the scene to one or more classes of objects by identifying the one or more classes of objects in the scene,
identify motion information of the scene with the classified objects based on the first RGB frame and the event data of the second RGB frame;
predict an object boundary of the second RGB frame using the one or more classes of objects in the scene; and
estimate the object boundary for the second RGB frame using the motion information.

17. The apparatus of claim 16 , wherein the processor is configured to:
identify spatially inconsistent regions in the scene in response to obtaining the event data by changing sensitivity levels of the event sensor, and
control to mask the identified spatially inconsistent regions in an image of the scene.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Saha Sujoy
• Srinivasan Karthik
• Yadav Sourabh Singh
• Palasamudram Suhas Shantaraja
• Peddigari Venkat Ramana
• Sk Pradeep Kumar
• Deshpande Pranav P
• Kar Akankshya

Classifications

• G06V40/25

  Recognition of walking or running movements, e.g. gait recognition · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06T7/12 Primary

  Edge-based segmentation · CPC title

• G06V20/13

  Satellite images · CPC title