Method and system of real-time image segmentation for image processing

Patent metadata

Field	Value
Publication number	US-10580140-B2
Application number	US-201615162478-A
Country	US
Kind code	B2
Filing date	May 23, 2016
Priority date	May 23, 2016
Publication date	Mar 3, 2020
Grant date	Mar 3, 2020

Abstract

Techniques related to a system, article, and method of real-time image segmentation for image processing.

First claim

What is claimed is:

1. A computer-implemented method of segmentation for image processing comprising:
obtaining image data of a plurality of frames of a video sequence comprising a prior frame and a subsequent frame relative to the prior frame and in display order of the video sequence;
using an accuracy-based segmentation algorithm to determine a prior contour separating segments on the prior frame; and
using a speed-based segmentation algorithm applied to the prior contour as taken from the prior frame and placed on the subsequent frame to determine a subsequent contour on the subsequent frame to be used to modify or display or both the subsequent frame.

2. The method of claim 1 comprising displaying the subsequent frame in real-time.

3. The method of claim 1 wherein the prior frame is displayed without using the prior contour formed to display the prior frame.

4. The method of claim 1 wherein the accuracy-based segmentation algorithm is slower and more accurate than the speed-based segmentation algorithm.

5. The method of claim 1 comprising locating the prior contour on the subsequent frame; and adjusting the prior contour to edges of at least one object on the subsequent frame to derive the subsequent contour.

6. The method of claim 1 wherein the subsequent contour is used to modify the image data of the subsequent frame to provide a visible difference to the subsequent frame.

7. The method of claim 1 wherein the accuracy-based segmentation algorithm is a graph-cut type of algorithm.

8. The method of claim 1 wherein the speed-based segmentation algorithm is an active-contour type of algorithm.

9. The method of claim 1 wherein the prior frame is not immediately prior to the subsequent frame.

10. The method of claim 1 wherein the prior frame is two frames before the subsequent frame in the display order.

11. The method of claim 1 wherein the accuracy-based segmentation algorithm is applied to an unknown region on the prior frame that separates one segment from another segment on the prior frame without applying the accuracy-based segmentation to the whole segments in order to provide the prior contour to be adjusted by the speed-based segmentation algorithm.

12. The method of claim 11 comprising generating the prior contour comprising obtaining a stored previous contour of an earlier frame earlier than the prior frame in display order, and applying the accuracy-based segmentation algorithm to the previous contour located on the prior frame.

13. The method of claim 1 :
the method comprising:
generating the prior contour comprising obtaining a stored previous contour of an earlier frame earlier than the prior frame in display order, and applying the accuracy-based segmentation algorithm to the previous contour located on the prior frame;
locating the prior contour on the subsequent frame; and
adjusting the prior contour to edges of at least one object on the subsequent frame to derive the subsequent contour; and
displaying the subsequent frame in real-time.

14. A computer-implemented system comprising:
at least one display;
at least one memory to store image data and at least one segmentation contour;
at least one processor communicatively coupled to the at least one of the memory and display; and
at least one segmentation unit operated by the at least one processor and to operate by:
obtaining image data of a plurality of frames of a video sequence comprising a prior frame and a subsequent frame relative to the prior frame and in display order of the video sequence;
using an accuracy-based segmentation algorithm to determine a prior contour separating segments on the prior frame; and
using a speed-based segmentation algorithm applied to the prior contour as taken from the prior frame and placed on the subsequent frame to determine a subsequent contour on the subsequent frame to be used to modify or display or both the subsequent frame.

15. The system of claim 14 wherein the at least one segmentation unit is to be operated by processing the segmentation of the frames of the video sequence on multiple threads feeding data to the at least one processor so that consecutive frames of the video sequence displayed in real-time are processed by different threads.

16. The system of claim 14 wherein the prior frame is at least two frames before the subsequent frame, wherein the at least one segmentation unit is to be operated by processing the segmentation of the frames of the video sequence on multiple threads feeding data to the at least one processor, and wherein a skipped frame that is skipped between the subsequent frame and the prior frame is displayed in real-time by a different thread than a thread that skipped the skipped frame.

17. The system of claim 14 wherein the prior contour of the prior frame used to form the subsequent contour is not used directly either to modify or display the prior frame.

18. The system of claim 14 wherein the at least one segmentation unit is to be operated by using one thread to use the prior contour of the prior frame as a preliminary contour to determine the subsequent contour to display the subsequent contour, and a different thread to use the prior frame as the subsequent frame to display the prior frame in real-time.

19. The system of claim 14 wherein the segmentation algorithms are arranged in an order so that the subsequent contour is a result of the processing on the subsequent frame used to convert the prior contour to the subsequent contour, and no subsequent contour is used to determine a contour of another frame.

20. The system of claim 14 wherein no two consecutive frame periods use the speed-based segmentation algorithm along the same thread.

21. The system of claim 14 wherein the accuracy-based segmentation algorithm is a graph-cut type of algorithm.

22. The system of claim 14 wherein the speed-based segmentation algorithm is an active-contour type of algorithm.

23. The system of claim 14
wherein the accuracy-based segmentation algorithm is a graph-cut type of algorithm; and
wherein the speed-based segmentation algorithm is an active-contour type of algorithm.

24. A non-transitory computer-readable medium having stored thereon instructions that when executed cause a computing device to operate by:
obtaining image data of a plurality of frames of a video sequence comprising a prior frame and a subsequent frame relative to the prior frame and in display order of the video sequence;
using an accuracy-based segmentation algorithm to determine a prior contour separating segments on the prior frame; and
using a speed-based segmentation algorithm applied to the prior contour as taken from the prior frame and placed on the subsequent frame to determine a subsequent contour on the subsequent frame to be used to modify or display or both the subsequent frame.

25. The non-transitory computer-readable medium of claim 24 wherein the accuracy-based segmentation algorithm is applied to an unknown region on the prior frame that separates one segment from another segment on the prior frame without applying the accuracy-based segmentation to the whole segments in order to provide the prior contour to be adjusted by the speed-based segmentation algorithm.

Assignees

• Intel Corp

Inventors

• Baruch Gilad

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20072

  Graph-based image processing · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/143 Primary

  involving probabilistic approaches, e.g. Markov random field [MRF] modelling · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title